diff options
160 files changed, 4778 insertions, 1376 deletions
diff --git a/.travis.yml b/.travis.yml index 3f07cc51..d2b60ed1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -11,10 +11,14 @@ python: - 3.3 - 3.4 - 3.5 - - pypy3 + - 3.6 + # disabled since setuptools dropped support for Python 3.0 - 3.2 + # - pypy3 + - pypy3.3-5.2-alpha1 before_install: - - if [[ $TRAVIS_PYTHON_VERSION == 2* || $TRAVIS_PYTHON_VERSION == 'pypy' ]]; then export F2B_PY_2=true && echo "Set F2B_PY_2"; fi - - if [[ $TRAVIS_PYTHON_VERSION == 3* || $TRAVIS_PYTHON_VERSION == 'pypy3' ]]; then export F2B_PY_3=true && echo "Set F2B_PY_3"; fi + - echo "running under $TRAVIS_PYTHON_VERSION" + - if [[ $TRAVIS_PYTHON_VERSION == 2* || $TRAVIS_PYTHON_VERSION == pypy* && $TRAVIS_PYTHON_VERSION != pypy3* ]]; then export F2B_PY_2=true && echo "Set F2B_PY_2"; fi + - if [[ $TRAVIS_PYTHON_VERSION == 3* || $TRAVIS_PYTHON_VERSION == pypy3* ]]; then export F2B_PY_3=true && echo "Set F2B_PY_3"; fi - travis_retry sudo apt-get update -qq # Set this so sudo executes the correct python binary # Anything not using sudo will already have the correct environment @@ -10,6 +10,141 @@ ver. 0.10.0 (2016/XX/XXX) - gonna-be-released-some-time-shining ----------- TODO: implementing of options resp. other tasks from PR #1346 + documentation should be extended (new options, etc) + +### Fixes +* `filter.d/apache-auth.conf`: + - better failure recognition using short form of regex (url/referer are foreign inputs, see gh-1645) +* `filter.d/apache-common.conf` (`filter.d/apache-*.conf`): + - support of apache log-format if logging into syslog/systemd (gh-1695), using parameter `logging`, + parameter usage for jail: + filter = apache-auth[logging=syslog] + parameter usage for `apache-common.local`: + logging = syslog +* `filter.d/pam-generic.conf`: + - [grave] injection on user name to host fixed +* `filter.d/sshd.conf`: + - rewritten using `prefregex` and used MLFID-related multi-line parsing + (by using tag `<F-MLFID>` instead of buffering with `maxlines`); + - optional parameter `mode` rewritten: normal (default), ddos, extra or aggressive (combines all), + see sshd for regex details) +* `filter.d/sendmail-reject.conf`: + - rewritten using `prefregex` and used MLFID-related multi-line parsing; + - optional parameter `mode` introduced: normal (default), extra or aggressive +* `filter.d/haproxy-http-auth`: do not mistake client port for part of an IPv6 address (gh-1745) +* `filter.d/postfix.conf`: + - updated to latest postfix formats + - joined several postfix filter together (normalized and optimized version, gh-1825) + - introduced new parameter `mode` (see gh-1825): more (default, combines normal and rbl), auth, normal, + rbl, ddos, extra or aggressive (combines all) + - postfix postscreen (resp. other RBL's compatibility fix, gh-1764, gh-1825) +* `filter.d/postfix-rbl.conf`: removed (replaced with `postfix[mode=rbl]`) +* `filter.d/postfix-sasl.conf`: removed (replaced with `postfix[mode=auth]`) +* `filter.d/roundcube-auth.conf`: + - fixed regex when `X-Real-IP` or/and `X-Forwarded-For` are present after host (gh-1303); + - fixed regex when logging authentication errors to journal instead to a local file (gh-1159); + - additionally fixed more complex injections on username (e. g. using dot after fake host). +* `filter.d/ejabberd-auth.conf`: fixed failregex - accept new log-format (gh-993) +* `action.d/complain.conf` + - fixed using new tag `<ip-rev>` (sh/dash compliant now) +* `action.d/sendmail-geoip-lines.conf` + - fixed using new tag `<ip-host>` (without external command execution) +* fail2ban-regex: fixed matched output by multi-line (buffered) parsing +* fail2ban-regex: support for multi-line debuggex URL implemented (gh-422) +* fixed ipv6-action errors on systems not supporting ipv6 and vice versa (gh-1741) +* fixed directory-based log-rotate for pyinotify-backend (gh-1778) + +### New Features +* New Actions: + +* New Filters: + +### Enhancements +* Introduced new filter option `prefregex` for pre-filtering using single regular expression (gh-1698); +* Many times faster and fewer CPU-hungry because of parsing with `maxlines=1`, so without + line buffering (scrolling of the buffer-window). + Combination of tags `<F-MLFID>` and `<F-NOFAIL>` can be used now to process multi-line logs + using single-line expressions: + - tag `<F-MLFID>`: used to identify resp. store failure info for groups of log-lines with the same + identifier (e. g. combined failure-info for the same conn-id by `<F-MLFID>(?:conn-id)</F-MLFID>`, + see sshd.conf for example); + - tag `<F-MLFFORGET>`: can be used as mark to forget current multi-line MLFID (e. g. by connection + closed, reset or disconnect etc); + - tag `<F-NOFAIL>`: used as mark for no-failure (helper to accumulate common failure-info, + e. g. from lines that contain IP-address); + Opposite to obsolete multi-line parsing (using buffering with `maxlines`) it is more precise and + can recognize multiple failure attempts within the same connection (MLFID). +* Several filters optimized with pre-filtering using new option `prefregex`, and multiline filter + using `<F-MLFID>` + `<F-NOFAIL>` combination; +* Exposes filter group captures in actions (non-recursive interpolation of tags `<F-...>`, + see gh-1698, gh-1110) +* Some filters extended with user name (can be used in gh-1243 to distinguish IP and user, + resp. to remove after success login the user-related failures only); +* Safer, more stable and faster replaceTag interpolation (switched from cycle over all tags + to re.sub with callable) +* substituteRecursiveTags optimization + moved in helpers facilities (because currently used + commonly in server and in client) +* New tags (usable in actions): + - `<fid>` - failure identifier (if raw resp. failures without IP address) + - `<ip-rev>` - PTR reversed representation of IP address + - `<ip-host>` - host name of the IP address + - `<F-...>` - interpolates to the corresponding filter group capture `...` + - `<fq-hostname>` - fully-qualified name of host (the same as `$(hostname -f)`) + - `<sh-hostname>` - short hostname (the same as `$(uname -n)`) +* Allow to use filter options by `fail2ban-regex`, example: + fail2ban-regex text.log "sshd[mode=aggressive]" +* Samples test case factory extended with filter options - dict in JSON to control + filter options (e. g. mode, etc.): + # filterOptions: {"mode": "aggressive"} +* Introduced new jail option "ignoreself", specifies whether the local resp. own IP addresses + should be ignored (default is true). Fail2ban will not ban a host which matches such addresses. + Option "ignoreip" affects additionally to "ignoreself" and don't need to include the DNS + resp. IPs of the host self. +* Regex will be compiled as MULTILINE only if needed (buffering with `maxlines` > 1), that enables: + - to improve performance by the single line parsing (see gh-1733); + - make regex more precise (because distinguish between anchors `^`/`$` for the begin/end of string + and the new-line character '\n', e. g. if coming from filters (like systemd journal) that allow + the parsing of log-entries contain new-line chars (as single entry); + - if multiline regex however expected (by single-line parsing without buffering) - prefix `(?m)` + could be used in regex to enable it; +* Implemented execution of `actionstart` on demand (conditional), if action depends on `family` (gh-1742): + - new action parameter `actionstart_on_demand` (bool) can be set to prevent/allow starting action + on demand (default retrieved automatically, if some conditional parameter `param?family=...` + presents in action properties), see `action.d/pf.conf` for example; + - additionally `actionstop` will be executed only for families previously executing `actionstart` + (starting on demand only) +* Introduced new command `actionflush`: executed in order to flush all bans at once + e. g. by unban all, reload with removing action, stop, shutdown the system (gh-1743), + the actions having `actionflush` do not execute `actionunban` for each single ticket +* Add new command `actionflush` default for several iptables/iptables-ipset actions (and common include); +* Add new jail option `logtimezone` to force the timezone on log lines that don't have an explicit one (gh-1773) +* Implemented zone abbreviations (like CET, CEST, etc.) and abbr+-offset functionality (accept zones + like 'CET+0100'), for the list of abbreviations see strptime.TZ_STR; +* Introduced new option `--timezone` (resp. `--TZ`) for `fail2ban-regex`. +* Tokens `%z` and `%Z` are changed (more precise now); +* Introduced new tokens `%Exz` and `%ExZ` that fully support zone abbreviations and/or offset-based + zones (implemented as enhancement using custom `datepattern`, because may be too dangerous for default + patterns and tokens like `%z`); + Note: the extended tokens supported zone abbreviations, but it can parse 1 or 3-5 char(s) in lowercase. + Don't use them in default date-patterns (if not anchored, few precise resp. optional). + Because python currently does not support mixing of case-sensitive with case-insensitive matching, + the TZ (in uppercase) cannot be combined with `%a`/`%b` etc (that are currently case-insensitive), + to avoid invalid date-time recognition in strings like '11-Aug-2013 03:36:11.372 error ...' with + wrong TZ "error". + Hence `%z` currently match literal Z|UTC|GMT only (and offset-based), and `%Exz` - all zone + abbreviations. +* `filter.d/courier-auth.conf`: support failed logins with method only +* Config reader's: introduced new syntax `%(section/option)s`, in opposite to extended interpolation of + python 3 `${section:option}` work with all supported python version in fail2ban and this syntax is + like our another features like `%(known/option)s`, etc. (gh-1750) +* Variable `default_backend` switched to `%(default/backend)s`, so totally backwards compatible now, + but now the setting of parameter `backend` in default section of `jail.local` can overwrite default + backend also (see gh-1750). In the future versions parameter `default_backend` can be removed (incompatibility, + possibly some distributions affected). + + +ver. 0.10.0-alpha-1 (2016/07/14) - ipv6-support-etc +----------- ### Fixes * [Grave] memory leak's fixed (gh-1277, gh-1234) @@ -83,6 +218,10 @@ TODO: implementing of options resp. other tasks from PR #1346 if configuration is clean (fails by wrong configured jails if option `-t` specified) * New command action parameter `actionrepair` - command executed in order to restore sane environment in error case of `actioncheck`. +* Reporting via abuseipdb.com: + - Bans can now be reported to abuseipdb + - Catagories must be set in the config + - Relevant log lines included in report ### Enhancements * Huge increasing of fail2ban performance and especially test-cases performance (see gh-1109) @@ -165,6 +304,31 @@ fail2ban-client set loglevel INFO - faster match and fewer searching of appropriate templates (DateDetector.matchTime calls rarer DateTemplate.matchDate now); - several standard filters extended with exact prefixed or anchored date templates; +* Added possibility to recognize restored state of the tickets (see gh-1669). + New option `norestored` introduced, to ignore restored tickets (after restart). + To avoid execution of ban/unban for the restored tickets, `norestored = true` + could be added in definition section of action. + For conditional usage in the shell-based actions an interpolation `<restored>` + could be used also. E. g. it is enough to add following script-piece at begin + of `actionban` (or `actionunban`) to prevent execution: + `if [ '<restored>' = '1' ]; then exit 0; fi;` + Several actions extended now using `norestored` option: + - complain.conf + - dshield.conf + - mail-buffered.conf + - mail-whois-lines.conf + - mail-whois.conf + - mail.conf + - sendmail-buffered.conf + - sendmail-geoip-lines.conf + - sendmail-whois-ipjailmatches.conf + - sendmail-whois-ipmatches.conf + - sendmail-whois-lines.conf + - sendmail-whois-matches.conf + - sendmail-whois.conf + - sendmail.conf + - smtp.py + - xarf-login-attack.conf * fail2ban-testcases: - `assertLogged` extended with parameter wait (to wait up to specified timeout, before we throw assert exception) + test cases rewritten using that @@ -172,29 +336,97 @@ fail2ban-client set loglevel INFO - new `with_foreground_server_thread` decorator to test several client/server commands -ver. 0.9.6 (2016/XX/XX) - wanna-be-released +ver. 0.9.8 (2016/XX/XXX) - wanna-be-released ----------- 0.9.x line is no longer heavily developed. If you are interested in new features (e.g. IPv6 support), please consider 0.10 branch and its releases. + +### Fixes +* Fix for systemd-backend: fail2ban hits the ulimit (out of file descriptors), see gh-991. + Partially back-ported from v.0.10. +* action.d/bsd-ipfw.conf + - Make the rule number, the action starts looking for a free slot to insert + the new rule, configurable (gh-1689) +* filter.d/apache-overflows.conf: + - Fixes resources greedy expression (see gh-1790); + - Rewritten without end-anchor ($), because of potential vulnerability on very long URLs. +* filter.d/asterisk.conf - fixed failregex AMI Asterisk authentification failed (see gh-1302) + +### New Features + +### Enhancements +* filter.d/kerio.conf - filter extended with new rules (see gh-1455) + + +ver. 0.9.7 (2017/05/11) - awaiting-victory +----------- + +### Fixes +* Fixed a systemd-journal handling in fail2ban-regex (gh-1657) +* filter.d/sshd.conf + - Fixed non-anchored part of failregex (misleading match of colon inside + IPv6 address instead of `: ` in the reason-part by missing space, gh-1658) + (0.10th resp. IPv6 relevant only, amend for gh-1479) +* config/pathes-freebsd.conf + - Fixed filenames for apache and nginx log files (gh-1667) +* filter.d/exim.conf + - optional part `(...)` after host-name before `[IP]` (gh-1751) + - new reason "Unrouteable address" for "rejected RCPT" regex (gh-1762) + - match of complex time like `D=2m42s` in regex "no MAIL in SMTP connection" (gh-1766) +* filter.d/sshd.conf + - new aggressive rules (gh-864): + - Connection reset by peer (multi-line rule during authorization process) + - No supported authentication methods available + - single line and multi-line expression optimized, added optional prefixes + and suffix (logged from several ssh versions), according to gh-1206; + - fixed expression received disconnect auth fail (optional space after port + part, gh-1652) + and suffix (logged from several ssh versions), according to gh-1206; +* filter.d/suhosin.conf + - greedy catch-all before `<HOST>` fixed (potential vulnerability) +* filter.d/cyrus-imap.conf + - accept entries without login-info resp. hostname before IP address (gh-1707) +* Filter tests extended with check of all config-regexp, that contains greedy catch-all + before `<HOST>`, that is hard-anchored at end or precise sub expression after `<HOST>` + +### New Features +* New Actions: + - action.d/netscaler: Block IPs on a Citrix Netscaler ADC (gh-1663) + +* New Filters: + - filter.d/domino-smtp: IBM Domino SMTP task (gh-1603) + +### Enhancements +* Introduced new log-level `MSG` (as INFO-2, equivalent to 18) + + +ver. 0.9.6 (2016/12/10) - stretch-is-coming +----------- + ### Fixes * Misleading add resp. enable of (already available) jail in database, that induced a subsequent error: last position of log file will be never retrieved (gh-795) * Fixed a distribution related bug within testReadStockJailConfForceEnabled (e.g. test-cases faults on Fedora, see gh-1353) -* Fixed pythonic filters and test scripts (running via wrong python version, +* Fixed pythonic filters and test scripts (running via wrong python version, uses "fail2ban-python" now); * Fixed test case "testSetupInstallRoot" for not default python version (also using direct call, out of virtualenv); * Fixed ambiguous wrong recognized date pattern resp. its optional parts (see gh-1512); * FIPS compliant, use sha1 instead of md5 if it not allowed (see gh-1540) * Monit config: scripting is not supported in path (gh-1556) +* `filter.d/apache-modsecurity.conf` + - Fixed for newer version (one space, gh-1626), optimized: non-greedy catch-all + replaced for safer match, unneeded catch-all anchoring removed, non-capturing * `filter.d/asterisk.conf` - Fixed to match different asterisk log prefix (source file: method:) +* `filter.d/dovecot.conf` + - Fixed failregex ignores failures through some not relevant info (gh-1623) * `filter.d/ignorecommands/apache-fakegooglebot` - - Fixed error within apache-fakegooglebot, that will be called + - Fixed error within apache-fakegooglebot, that will be called with wrong python version (gh-1506) * `filter.d/assp.conf` - Extended failregex and test cases to handle ASSP V1 and V2 (gh-1494) @@ -208,18 +440,21 @@ releases. - recognized "Failed publickey for" (gh-1477); - optimized failregex to match all of "Failed any-method for ... from <HOST>" (gh-1479) - eliminated possible complex injections (on user-name resp. auth-info, see gh-1479) - + - optional port part after host (see gh-1533, gh-1581) ### New Features * New Actions: - `action.d/npf.conf` for NPF, the latest packet filter for NetBSD +* New Filters: + - `filter.d/mongodb-auth.conf` for MongoDB (document-oriented NoSQL database engine) + (gh-1586, gh-1606 and gh-1607) ### Enhancements -* DateTemplate regexp extended with the word-end boundary, additionally to +* DateTemplate regexp extended with the word-end boundary, additionally to word-start boundary -* Introduces new command "fail2ban-python", as automatically created symlink to +* Introduces new command "fail2ban-python", as automatically created symlink to python executable, where fail2ban currently installed (resp. its modules are located): - - allows to use the same version, fail2ban currently running, e.g. in + - allows to use the same version, fail2ban currently running, e.g. in external scripts just via replace python with fail2ban-python: ```diff -#!/usr/bin/env python @@ -3,6 +3,7 @@ bin/fail2ban-regex bin/fail2ban-server bin/fail2ban-testcases ChangeLog +config/action.d/abuseipdb.conf config/action.d/apf.conf config/action.d/badips.conf config/action.d/badips.py @@ -19,6 +20,7 @@ config/action.d/firewallcmd-multiport.conf config/action.d/firewallcmd-new.conf config/action.d/firewallcmd-rich-logging.conf config/action.d/firewallcmd-rich-rules.conf +config/action.d/helpers-common.conf config/action.d/hostsdeny.conf config/action.d/ipfilter.conf config/action.d/ipfw.conf @@ -38,9 +40,11 @@ config/action.d/mail-whois-common.conf config/action.d/mail-whois.conf config/action.d/mail-whois-lines.conf config/action.d/mynetwatchman.conf +config/action.d/netscaler.conf config/action.d/nftables-allports.conf config/action.d/nftables-common.conf config/action.d/nftables-multiport.conf +config/action.d/npf.conf config/action.d/nsupdate.conf config/action.d/osx-afctl.conf config/action.d/osx-ipfw.conf @@ -83,6 +87,7 @@ config/filter.d/courier-auth.conf config/filter.d/courier-smtp.conf config/filter.d/cyrus-imap.conf config/filter.d/directadmin.conf +config/filter.d/domino-smtp.conf config/filter.d/dovecot.conf config/filter.d/dropbear.conf config/filter.d/drupal-auth.conf @@ -100,6 +105,7 @@ config/filter.d/horde.conf config/filter.d/ignorecommands/apache-fakegooglebot config/filter.d/kerio.conf config/filter.d/lighttpd-auth.conf +config/filter.d/mongodb-auth.conf config/filter.d/monit.conf config/filter.d/murmur.conf config/filter.d/mysqld-auth.conf @@ -117,8 +123,6 @@ config/filter.d/perdition.conf config/filter.d/php-url-fopen.conf config/filter.d/portsentry.conf config/filter.d/postfix.conf -config/filter.d/postfix-rbl.conf -config/filter.d/postfix-sasl.conf config/filter.d/proftpd.conf config/filter.d/pure-ftpd.conf config/filter.d/qmail.conf @@ -136,7 +140,6 @@ config/filter.d/solid-pop3d.conf config/filter.d/squid.conf config/filter.d/squirrelmail.conf config/filter.d/sshd.conf -config/filter.d/sshd-ddos.conf config/filter.d/stunnel.conf config/filter.d/suhosin.conf config/filter.d/tine20.conf @@ -146,6 +149,7 @@ config/filter.d/webmin-auth.conf config/filter.d/wuftpd.conf config/filter.d/xinetd-fail.conf config/jail.conf +config/paths-arch.conf config/paths-common.conf config/paths-debian.conf config/paths-fedora.conf @@ -154,6 +158,7 @@ config/paths-opensuse.conf config/paths-osx.conf CONTRIBUTING.md COPYING +.coveragerc DEVELOP fail2ban-2to3 fail2ban/client/actionreader.py @@ -217,6 +222,7 @@ fail2ban/tests/config/filter.d/simple.conf fail2ban/tests/config/filter.d/test.conf fail2ban/tests/config/filter.d/test.local fail2ban/tests/config/filter.d/zzz-generic-example.conf +fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf fail2ban/tests/config/jail.conf fail2ban/tests/databasetestcase.py fail2ban/tests/datedetectortestcase.py @@ -272,6 +278,7 @@ fail2ban/tests/files/logs/courier-auth fail2ban/tests/files/logs/courier-smtp fail2ban/tests/files/logs/cyrus-imap fail2ban/tests/files/logs/directadmin +fail2ban/tests/files/logs/domino-smtp fail2ban/tests/files/logs/dovecot fail2ban/tests/files/logs/dropbear fail2ban/tests/files/logs/drupal-auth @@ -287,6 +294,7 @@ fail2ban/tests/files/logs/haproxy-http-auth fail2ban/tests/files/logs/horde fail2ban/tests/files/logs/kerio fail2ban/tests/files/logs/lighttpd-auth +fail2ban/tests/files/logs/mongodb-auth fail2ban/tests/files/logs/monit fail2ban/tests/files/logs/murmur fail2ban/tests/files/logs/mysqld-auth @@ -304,8 +312,6 @@ fail2ban/tests/files/logs/perdition fail2ban/tests/files/logs/php-url-fopen fail2ban/tests/files/logs/portsentry fail2ban/tests/files/logs/postfix -fail2ban/tests/files/logs/postfix-rbl -fail2ban/tests/files/logs/postfix-sasl fail2ban/tests/files/logs/proftpd fail2ban/tests/files/logs/pure-ftpd fail2ban/tests/files/logs/qmail @@ -322,7 +328,6 @@ fail2ban/tests/files/logs/solid-pop3d fail2ban/tests/files/logs/squid fail2ban/tests/files/logs/squirrelmail fail2ban/tests/files/logs/sshd -fail2ban/tests/files/logs/sshd-ddos fail2ban/tests/files/logs/stunnel fail2ban/tests/files/logs/suhosin fail2ban/tests/files/logs/tine20 @@ -332,6 +337,8 @@ fail2ban/tests/files/logs/webmin-auth fail2ban/tests/files/logs/wuftpd fail2ban/tests/files/logs/xinetd-fail fail2ban/tests/files/logs/zzz-generic-example +fail2ban/tests/files/logs/zzz-sshd-obsolete-multiline +fail2ban/tests/files/testcase01a.log fail2ban/tests/files/testcase01.log fail2ban/tests/files/testcase02.log fail2ban/tests/files/testcase03.log @@ -340,6 +347,7 @@ fail2ban/tests/files/testcase-journal.log fail2ban/tests/files/testcase-multiline.log fail2ban/tests/files/testcase-usedns.log fail2ban/tests/files/testcase-wrong-char.log +fail2ban/tests/files/zzz-sshd-obsolete-multiline.log fail2ban/tests/filtertestcase.py fail2ban/tests/__init__.py fail2ban/tests/misctestcase.py @@ -386,6 +394,7 @@ man/fail2ban-testcases.1 man/fail2ban-testcases.h2m man/generate-man man/jail.conf.5 +.pylintrc README.md README.Solaris RELEASE @@ -17,9 +17,13 @@ Though Fail2Ban is able to reduce the rate of incorrect authentications attempts, it cannot eliminate the risk that weak authentication presents. Configure services to use only two factor or public/private authentication mechanisms if you really want to protect services. + +<img src="http://www.worldipv6launch.org/wp-content/themes/ipv6/downloads/World_IPv6_launch_logo.svg" height="52pt"/> | Since v0.10 fail2ban supports the matching of the IPv6 addresses. +------|------ This README is a quick introduction to Fail2ban. More documentation, FAQ, HOWTOs -are available in fail2ban(1) manpage and on the website http://www.fail2ban.org +are available in fail2ban(1) manpage, [Wiki](https://github.com/fail2ban/fail2ban/wiki) +and on the website http://www.fail2ban.org Installation: ------------- @@ -86,7 +90,7 @@ Contact: See [CONTRIBUTING.md](https://github.com/fail2ban/fail2ban/blob/master/CONTRIBUTING.md) ### You just appreciate this program: -send kudos to the original author ([Cyril Jaquier](mailto: Cyril Jaquier <cyril.jaquier@fail2ban.org>)) +send kudos to the original author ([Cyril Jaquier](mailto:cyril.jaquier@fail2ban.org)) or *better* to the [mailing list](https://lists.sourceforge.net/lists/listinfo/fail2ban-users) since Fail2Ban is "community-driven" for years now. @@ -53,7 +53,7 @@ Preparation or an alternative for comparison with previous release - git diff 0.9.5 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST + git diff 0.10.0 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST sort MANIFEST | uniq | sponge MANIFEST * Run:: @@ -70,7 +70,7 @@ Preparation * clean up current directory:: - diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.5/ + diff -rul --exclude \*.pyc . /tmp/fail2ban-0.10.0/ * Only differences should be files that you don't want distributed. @@ -83,7 +83,7 @@ Preparation * To generate a list of committers use e.g.:: - git shortlog -sn 0.9.5.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' + git shortlog -sn 0.10.0.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' * Ensure the top of the ChangeLog has the right version and current date. * Ensure the top entry of the ChangeLog has the right version and current date. @@ -106,7 +106,7 @@ Preparation * Tag the release by using a signed (and annotated) tag. Cut/paste release ChangeLog entry as tag annotation:: - git tag -s 0.9.5 + git tag -s 0.10.0 Pre Release =========== @@ -16,6 +16,7 @@ Alexander Koeppe (IPv6 support) Alexandre Perrin (kAworu) Amir Caspi Amy +Andrew James Collett (ajcollett) Andrew St. Jean Andrey G. Grozin Andy Fragen @@ -60,6 +61,7 @@ John Thoe Jacques Lav!gnotte Johannes Weberhofer Jason H Martin +Jeaye Wilkerson Jisoo Park Joel M Snyder Jonathan Kamens @@ -111,6 +113,7 @@ Sean DuBois Sebastian Arcus Serg G. Brester Sergey Safarov +Shaun C. Sireyessire silviogarbes Stefan Tatschner @@ -121,6 +124,7 @@ Thomas Mayer Tom Pike Tom Hendrikx Tomas Pihl +Thomas Skierlo (phaleas) Tony Lawrence Tomasz Ciolek Tyler diff --git a/config/action.d/abuseipdb.conf b/config/action.d/abuseipdb.conf new file mode 100644 index 00000000..15e41fbe --- /dev/null +++ b/config/action.d/abuseipdb.conf @@ -0,0 +1,105 @@ +# Fail2ban configuration file +# +# Action to report IP address to abuseipdb.com +# You must sign up to obtain an API key from abuseipdb.com. +# +# NOTE: These reports may include sensitive Info. +# If you want cleaner reports that ensure no user data see the helper script at the below website. +# +# IMPORTANT: +# +# Reporting an IP of abuse is a serious complaint. Make sure that it is +# serious. Fail2ban developers and network owners recommend you only use this +# action for: +# * The recidive where the IP has been banned multiple times +# * Where maxretry has been set quite high, beyond the normal user typing +# password incorrectly. +# * For filters that have a low likelihood of receiving human errors +# +# This action relies on a api_key being added to the above action conf, +# and the appropriate categories set. +# +# Example, for ssh bruteforce (in section [sshd] of `jail.local`): +# action = %(known/action)s +# %(action_abuseipdb)s[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"] +# +# See below for catagories. +# +# Original Ref: https://wiki.shaunc.com/wikka.php?wakka=ReportingToAbuseIPDBWithFail2Ban +# Added to fail2ban by Andrew James Collett (ajcollett) + +## abuseIPDB Catagories, `the abuseipdb_category` MUST be set in the jail.conf action call. +# Example, for ssh bruteforce: action = %(action_abuseipdb)s[abuseipdb_category="18,22"] +# ID Title Description +# 3 Fraud Orders +# 4 DDoS Attack +# 9 Open Proxy +# 10 Web Spam +# 11 Email Spam +# 14 Port Scan +# 18 Brute-Force +# 19 Bad Web Bot +# 20 Exploited Host +# 21 Web App Attack +# 22 SSH Secure Shell (SSH) abuse. Use this category in combination with more specific categories. +# 23 IoT Targeted +# See https://abuseipdb.com/categories for more descriptions + +[Definition] + +# Option: actionstart +# Notes.: command executed once at the start of Fail2Ban. +# Values: CMD +# +actionstart = + +# Option: actionstop +# Notes.: command executed once at the end of Fail2Ban +# Values: CMD +# +actionstop = + +# Option: actioncheck +# Notes.: command executed once before each actionban command +# Values: CMD +# +actioncheck = + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# +# ** IMPORTANT! ** +# +# By default, this posts directly to AbuseIPDB's API, unfortunately +# this results in a lot of backslashes/escapes appearing in the +# reports. This also may include info like your hostname. +# If you have your own web server with PHP available, you can +# use my (Shaun's) helper PHP script by commenting out the first #actionban +# line below, uncommenting the second one, and pointing the URL at +# wherever you install the helper script. For the PHP helper script, see +# <https://wiki.shaunc.com/wikka.php?wakka=ReportingToAbuseIPDBWithFail2Ban> +# +# --ciphers ecdhe_ecdsa_aes_256_sha is used to workaround a +# "NSS error -12286" from curl as it attempts to connect using +# SSLv3. See https://www.centos.org/forums/viewtopic.php?t=52732 +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionban = curl --fail --ciphers ecdhe_ecdsa_aes_256_sha --data 'key=<abuseipdb_apikey>' --data-urlencode 'comment=<matches>' --data 'ip=<ip>' --data 'category=<abuseipdb_category>' "https://www.abuseipdb.com/report/json" + +# Option: actionunban +# Notes.: command executed when unbanning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionunban = + +[Init] +# Option: abuseipdb_apikey +# Notes Your API key from abuseipdb.com +# Values: STRING Default: None +# Register for abuseipdb [https://www.abuseipdb.com], get api key and set below. +# You will need to set the catagory in the action call. +abuseipdb_apikey = diff --git a/config/action.d/bsd-ipfw.conf b/config/action.d/bsd-ipfw.conf index 8b0a51aa..65a5e39b 100644 --- a/config/action.d/bsd-ipfw.conf +++ b/config/action.d/bsd-ipfw.conf @@ -14,7 +14,7 @@ # Notes.: command executed once at the start of Fail2Ban. # Values: CMD # -actionstart = ipfw show | fgrep -q 'table(<table>)' || ( ipfw show | awk 'BEGIN { b = 1 } { if ($1 <= b) { b = $1 + 1 } else { e = b } } END { if (e) exit e <br> else exit b }'; num=$?; ipfw -q add $num <blocktype> <block> from table\(<table>\) to me <port>; echo $num > "<startstatefile>" ) +actionstart = ipfw show | fgrep -q 'table(<table>)' || ( ipfw show | awk 'BEGIN { b = <lowest_rule_num> } { if ($1 < b) {} else if ($1 == b) { b = $1 + 1 } else { e = b } } END { if (e) exit e <br> else exit b }'; num=$?; ipfw -q add $num <blocktype> <block> from table\(<table>\) to me <port>; echo $num > "<startstatefile>" ) # Option: actionstop @@ -81,3 +81,11 @@ block = ip # Values: STRING # blocktype = unreach port + +# Option: lowest_rule_num +# Notes: When fail2ban starts with action and there is no rule for the given table yet +# then fail2ban will start looking for an empty slot starting with this rule number. +# Values: NUM +lowest_rule_num = 111 + + diff --git a/config/action.d/complain.conf b/config/action.d/complain.conf index e4ceb35f..1f74d635 100644 --- a/config/action.d/complain.conf +++ b/config/action.d/complain.conf @@ -34,6 +34,12 @@ before = helpers-common.conf [Definition] +# Used in test cases for coverage internal transformations +debug = 0 + +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -58,9 +64,11 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = oifs=${IFS}; - IFS=.; SEP_IP=( <ip> ); set -- ${SEP_IP}; ADDRESSES=$(dig +short -t txt -q $4.$3.$2.$1.abuse-contacts.abusix.org); - IFS=,; ADDRESSES=$(echo $ADDRESSES) +actionban = oifs=${IFS}; + RESOLVER_ADDR="%(addr_resolver)s" + if [ "<debug>" -gt 0 ]; then echo "try to resolve $RESOLVER_ADDR"; fi + ADDRESSES=$(dig +short -t txt -q $RESOLVER_ADDR | tr -d '"') + IFS=,; ADDRESSES=$(echo $ADDRESSES) IFS=${oifs} IP=<ip> if [ ! -z "$ADDRESSES" ]; then @@ -78,7 +86,12 @@ actionban = oifs=${IFS}; # actionunban = -[Init] +# Server as resolver used in dig command +# +addr_resolver = <ip-rev>abuse-contacts.abusix.org + +# Default message used for abuse content +# message = Dear Sir/Madam,\n\nWe have detected abuse from the IP address $IP, which according to a abusix.com is on your network. We would appreciate if you would investigate and take action as appropriate.\n\nLog lines are given below, but please ask if you require any further information.\n\n(If you are not the correct person to contact about this please accept our apologies - your e-mail address was extracted from the whois record by an automated process.)\n\n This mail was generated by Fail2Ban.\nThe recipient address of this report was provided by the Abuse Contact DB by abusix.com. abusix.com does not maintain the content of the database. All information which we pass out, derives from the RIR databases and is processed for ease of use. If you want to change or report non working abuse contacts please contact the appropriate RIR. If you have any further question, contact abusix.com directly via email (info@abusix.com). Information about the Abuse Contact Database can be found here: https://abusix.com/global-reporting/abuse-contact-db\nabusix.com is neither responsible nor liable for the content or accuracy of this message.\n # Path to the log files which contain relevant lines for the abuser IP diff --git a/config/action.d/dshield.conf b/config/action.d/dshield.conf index a0041986..4f2e09ca 100644 --- a/config/action.d/dshield.conf +++ b/config/action.d/dshield.conf @@ -28,6 +28,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD diff --git a/config/action.d/dummy.conf b/config/action.d/dummy.conf index dc4e1dbf..41250c27 100644 --- a/config/action.d/dummy.conf +++ b/config/action.d/dummy.conf @@ -10,14 +10,23 @@ # Notes.: command executed once at the start of Fail2Ban. # Values: CMD # -actionstart = touch /var/run/fail2ban/fail2ban.dummy - printf %%b "<init>\n" >> /var/run/fail2ban/fail2ban.dummy +actionstart = if [ ! -z '<target>' ]; then touch <target>; fi; + printf %%b "<init>\n" <to_target> + echo "%(debug)s started" + +# Option: actionflush +# Notes.: command executed once to flush (clear) all IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = printf %%b "-*\n" <to_target> + echo "%(debug)s clear all" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD # -actionstop = rm -f /var/run/fail2ban/fail2ban.dummy +actionstop = if [ ! -z '<target>' ]; then rm -f <target>; fi; + echo "%(debug)s stopped" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -31,7 +40,8 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "+<ip>\n" >> /var/run/fail2ban/fail2ban.dummy +actionban = printf %%b "+<ip>\n" <to_target> + echo "%(debug)s banned <ip> (family: <family>)" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -39,9 +49,15 @@ actionban = printf %%b "+<ip>\n" >> /var/run/fail2ban/fail2ban.dummy # Tags: See jail.conf(5) man page # Values: CMD # -actionunban = printf %%b "-<ip>\n" >> /var/run/fail2ban/fail2ban.dummy +actionunban = printf %%b "-<ip>\n" <to_target> + echo "%(debug)s unbanned <ip> (family: <family>)" + + +debug = [<name>] <actname> <target> -- [Init] init = 123 +target = /var/run/fail2ban/fail2ban.dummy +to_target = >> <target> diff --git a/config/action.d/firewallcmd-rich-logging.conf b/config/action.d/firewallcmd-rich-logging.conf index d2c8fc2f..badfee83 100644 --- a/config/action.d/firewallcmd-rich-logging.conf +++ b/config/action.d/firewallcmd-rich-logging.conf @@ -35,7 +35,7 @@ actioncheck = # service name example: # firewall-cmd --zone=<zone> --add-rich-rule="rule family='<family>' source address='<ip>' service name='<service>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>" # -# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges seperated by a comma or space for an example: http, https, 22-60, 18 smtp +# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"; done diff --git a/config/action.d/firewallcmd-rich-rules.conf b/config/action.d/firewallcmd-rich-rules.conf index e64c3823..bed71797 100644 --- a/config/action.d/firewallcmd-rich-rules.conf +++ b/config/action.d/firewallcmd-rich-rules.conf @@ -33,7 +33,7 @@ actioncheck = # service name example: # firewall-cmd --zone=<zone> --add-rich-rule="rule family='ipv4' source address='<ip>' service name='<service>' <rich-blocktype>" # -# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges seperated by a comma or space for an example: http, https, 22-60, 18 smtp +# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' <rich-blocktype>"; done diff --git a/config/action.d/helpers-common.conf b/config/action.d/helpers-common.conf index 7fa8e9e4..5799d9d3 100644 --- a/config/action.d/helpers-common.conf +++ b/config/action.d/helpers-common.conf @@ -5,9 +5,12 @@ # (printf %%b "Log-excerpt contains 'test':\n"; %(_grep_logs)s; printf %%b "Log-excerpt contains 'test':\n") | mail ...
#
_grep_logs = logpath="<logpath>"; grep <grepopts> -E %(_grep_logs_args)s $logpath | <greplimit>
-_grep_logs_args = '(^|[^0-9])<ip>([^0-9]|$)'
+_grep_logs_args = "(^|[^0-9a-fA-F:])$(echo '<ip>' | sed 's/\./\\./g')([^0-9a-fA-F:]|$)"
+
+# Used for actions, that should not by executed if ticket was restored:
+_bypass_if_restored = if [ '<restored>' = '1' ]; then exit 0; fi;
[Init]
greplimit = tail -n <grepmax>
grepmax = 1000
-grepopts = -m <grepmax>
\ No newline at end of file +grepopts = -m <grepmax>
diff --git a/config/action.d/iptables-allports.conf b/config/action.d/iptables-allports.conf index 15f3cbcc..dbea5984 100644 --- a/config/action.d/iptables-allports.conf +++ b/config/action.d/iptables-allports.conf @@ -26,7 +26,7 @@ actionstart = <iptables> -N f2b-<name> # Values: CMD # actionstop = <iptables> -D <chain> -p <protocol> -j f2b-<name> - <iptables> -F f2b-<name> + <actionflush> <iptables> -X f2b-<name> # Option: actioncheck diff --git a/config/action.d/iptables-common.conf b/config/action.d/iptables-common.conf index a3921021..e016ef2f 100644 --- a/config/action.d/iptables-common.conf +++ b/config/action.d/iptables-common.conf @@ -16,6 +16,14 @@ after = iptables-blocktype.local iptables-common.local # iptables-blocktype.local is obsolete +[Definition] + +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = <iptables> -F f2b-<name> + [Init] diff --git a/config/action.d/iptables-ipset-proto4.conf b/config/action.d/iptables-ipset-proto4.conf index 2f63cd4b..30353f36 100644 --- a/config/action.d/iptables-ipset-proto4.conf +++ b/config/action.d/iptables-ipset-proto4.conf @@ -30,12 +30,19 @@ before = iptables-common.conf actionstart = ipset --create f2b-<name> iphash <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype> + +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = ipset --flush f2b-<name> + # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD # actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype> - ipset --flush f2b-<name> + <actionflush> ipset --destroy f2b-<name> # Option: actionban diff --git a/config/action.d/iptables-ipset-proto6-allports.conf b/config/action.d/iptables-ipset-proto6-allports.conf index 113f599e..b761ad8c 100644 --- a/config/action.d/iptables-ipset-proto6-allports.conf +++ b/config/action.d/iptables-ipset-proto6-allports.conf @@ -29,12 +29,18 @@ before = iptables-common.conf actionstart = ipset create <ipmset> hash:ip timeout <bantime><familyopt> <iptables> -I <chain> -m set --match-set <ipmset> src -j <blocktype> +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = ipset flush <ipmset> + # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD # actionstop = <iptables> -D <chain> -m set --match-set <ipmset> src -j <blocktype> - ipset flush <ipmset> + <actionflush> ipset destroy <ipmset> # Option: actionban diff --git a/config/action.d/iptables-ipset-proto6.conf b/config/action.d/iptables-ipset-proto6.conf index dee7b029..e337eedf 100644 --- a/config/action.d/iptables-ipset-proto6.conf +++ b/config/action.d/iptables-ipset-proto6.conf @@ -29,12 +29,18 @@ before = iptables-common.conf actionstart = ipset create <ipmset> hash:ip timeout <bantime><familyopt> <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype> +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = ipset flush <ipmset> + # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD # actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype> - ipset flush <ipmset> + <actionflush> ipset destroy <ipmset> # Option: actionban diff --git a/config/action.d/iptables-multiport-log.conf b/config/action.d/iptables-multiport-log.conf index 1777ce62..62c2b4b1 100644 --- a/config/action.d/iptables-multiport-log.conf +++ b/config/action.d/iptables-multiport-log.conf @@ -26,13 +26,19 @@ actionstart = <iptables> -N f2b-<name> <iptables> -I f2b-<name>-log -j LOG --log-prefix "$(expr f2b-<name> : '\(.\{1,23\}\)'):DROP " --log-level warning -m limit --limit 6/m --limit-burst 2 <iptables> -A f2b-<name>-log -j <blocktype> +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = <iptables> -F f2b-<name> + <iptables> -F f2b-<name>-log + # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD # actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name> - <iptables> -F f2b-<name> - <iptables> -F f2b-<name>-log + <actionflush> <iptables> -X f2b-<name> <iptables> -X f2b-<name>-log diff --git a/config/action.d/iptables-multiport.conf b/config/action.d/iptables-multiport.conf index 9fd87d20..c05f6ffc 100644 --- a/config/action.d/iptables-multiport.conf +++ b/config/action.d/iptables-multiport.conf @@ -23,7 +23,7 @@ actionstart = <iptables> -N f2b-<name> # Values: CMD # actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name> - <iptables> -F f2b-<name> + <actionflush> <iptables> -X f2b-<name> # Option: actioncheck diff --git a/config/action.d/iptables-new.conf b/config/action.d/iptables-new.conf index 795bc601..5b316807 100644 --- a/config/action.d/iptables-new.conf +++ b/config/action.d/iptables-new.conf @@ -25,7 +25,7 @@ actionstart = <iptables> -N f2b-<name> # Values: CMD # actionstop = <iptables> -D <chain> -m state --state NEW -p <protocol> --dport <port> -j f2b-<name> - <iptables> -F f2b-<name> + <actionflush> <iptables> -X f2b-<name> # Option: actioncheck diff --git a/config/action.d/iptables-xt_recent-echo.conf b/config/action.d/iptables-xt_recent-echo.conf index 018d2cf6..1970de14 100644 --- a/config/action.d/iptables-xt_recent-echo.conf +++ b/config/action.d/iptables-xt_recent-echo.conf @@ -35,6 +35,12 @@ before = iptables-common.conf # shorter of the two timeouts actually matters. actionstart = if [ `id -u` -eq 0 ];then <iptables> -I <chain> -m recent --update --seconds 3600 --name <iptname> -j <blocktype>;fi +# Option: actionflush +# +# [TODO] Flushing is currently not implemented for xt_recent +# +actionflush = + # Option: actionstop # Notes.: command executed once at the end of Fail2Ban # Values: CMD diff --git a/config/action.d/iptables.conf b/config/action.d/iptables.conf index 38985ffa..bf83e24a 100644 --- a/config/action.d/iptables.conf +++ b/config/action.d/iptables.conf @@ -23,7 +23,7 @@ actionstart = <iptables> -N f2b-<name> # Values: CMD # actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name> - <iptables> -F f2b-<name> + <actionflush> <iptables> -X f2b-<name> # Option: actioncheck diff --git a/config/action.d/mail-buffered.conf b/config/action.d/mail-buffered.conf index 914d4a5a..88cd623f 100644 --- a/config/action.d/mail-buffered.conf +++ b/config/action.d/mail-buffered.conf @@ -6,6 +6,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -14,7 +17,7 @@ actionstart = printf %%b "Hi,\n The jail <name> has been started successfully.\n Output will be buffered until <lines> lines are available.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: started on `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest> # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -25,13 +28,13 @@ actionstop = if [ -f <tmpfile> ]; then These hosts have been banned by Fail2Ban.\n `cat <tmpfile>` Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: Summary from `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: Summary from <fq-hostname>" <dest> rm <tmpfile> fi printf %%b "Hi,\n The jail <name> has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest> # Option: actioncheck # Notes.: command executed once before each actionban command diff --git a/config/action.d/mail-whois-lines.conf b/config/action.d/mail-whois-lines.conf index cbd970c9..37e2d9b0 100644 --- a/config/action.d/mail-whois-lines.conf +++ b/config/action.d/mail-whois-lines.conf @@ -11,6 +11,9 @@ before = mail-whois-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -18,7 +21,7 @@ before = mail-whois-common.conf actionstart = printf %%b "Hi,\n The jail <name> has been started successfully.\n Regards,\n - Fail2Ban" | <mailcmd> -s "[Fail2Ban] <name>: started on `uname -n`" <dest> + Fail2Ban" | <mailcmd> "[Fail2Ban] <name>: started on <fq-hostname>" <dest> # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -27,7 +30,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail <name> has been stopped.\n Regards,\n - Fail2Ban" | <mailcmd> -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest> + Fail2Ban" | <mailcmd> "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest> # Option: actioncheck # Notes.: command executed once before each actionban command @@ -52,7 +55,8 @@ _ban_mail_content = ( printf %%b "Hi,\n printf %%b "\n Regards,\n Fail2Ban" ) -actionban = %(_ban_mail_content)s | <mailcmd> "[Fail2Ban] <name>: banned <ip> from `uname -n`" <dest> + +actionban = %(_ban_mail_content)s | <mailcmd> "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest> # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the diff --git a/config/action.d/mail-whois.conf b/config/action.d/mail-whois.conf index 018c327d..1f69f4c6 100644 --- a/config/action.d/mail-whois.conf +++ b/config/action.d/mail-whois.conf @@ -10,6 +10,9 @@ before = mail-whois-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -17,7 +20,7 @@ before = mail-whois-common.conf actionstart = printf %%b "Hi,\n The jail <name> has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: started on `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest> # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -26,7 +29,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail <name> has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest> # Option: actioncheck # Notes.: command executed once before each actionban command @@ -46,7 +49,7 @@ actionban = printf %%b "Hi,\n Here is more information about <ip> :\n `%(_whois_command)s`\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest> # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the diff --git a/config/action.d/mail.conf b/config/action.d/mail.conf index 7bf51a1d..cfc1cf65 100644 --- a/config/action.d/mail.conf +++ b/config/action.d/mail.conf @@ -6,6 +6,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -13,7 +16,7 @@ actionstart = printf %%b "Hi,\n The jail <name> has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: started on `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: started on <fq-hostname>" <dest> # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -22,7 +25,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail <name> has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: stopped on <fq-hostname>" <dest> # Option: actioncheck # Notes.: command executed once before each actionban command @@ -40,7 +43,7 @@ actionban = printf %%b "Hi,\n The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from `uname -n`" <dest> + Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip> from <fq-hostname>" <dest> # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the diff --git a/config/action.d/netscaler.conf b/config/action.d/netscaler.conf new file mode 100644 index 00000000..87f7e7bf --- /dev/null +++ b/config/action.d/netscaler.conf @@ -0,0 +1,33 @@ +# Fail2ban Citrix Netscaler Action +# by Juliano Jeziorny +# juliano@jeziorny.eu +# +# The script will add offender IPs to a dataset on netscaler, the dataset can then be used to block the IPs at a cs/vserver or global level +# This dataset is then used to block IPs using responder policies on the netscaler. +# +# The script assumes using HTTPS with unsecure certificate to access the netscaler, +# if you have a valid certificate installed remove the -k from the curl lines, or if you want http change it accordingly (and remove the -k) +# +# This action depends on curl +# +# You need to populate the 3 options inside Init +# +# ns_host: IP or hostname of netslcaer appliance +# ns_auth: username:password, suggest base64 encoded for a little added security (echo -n "username:password" | base64) +# ns_dataset: Name of the netscaler dataset holding the IPs to be blocked. +# +# For further details on how to use it please check http://blog.ckzone.eu/2017/01/fail2ban-action-for-citrix-netscaler.html + +[Init] +ns_host = +ns_auth = +ns_dataset = + +[Definition] +actionstart = curl -kH 'Authorization: Basic <ns_auth>' https://<ns_host>/nitro/v1/config + +actioncheck = + +actionban = curl -k -H 'Authorization: Basic <ns_auth>' -X PUT -d '{"policydataset_value_binding":{"name":"<ns_dataset>","value":"<ip>"}}' https://<ns_host>/nitro/v1/config/ + +actionunban = curl -H 'Authorization: Basic <ns_auth>' -X DELETE -k "https://<ns_host>/nitro/v1/config/policydataset_value_binding/<ns_dataset>?args=value:<ip>" diff --git a/config/action.d/pf.conf b/config/action.d/pf.conf index b7476fa2..deb38c09 100644 --- a/config/action.d/pf.conf +++ b/config/action.d/pf.conf @@ -18,6 +18,9 @@ actionstart = echo "table <<tablename>-<name>> persist counters" | pfctl -f- echo "block proto <protocol> from <<tablename>-<name>> to <actiontype>" | pfctl -f- +# Option: start_on_demand - to start action on demand +# Example: `action=pf[actionstart_on_demand=true]` +actionstart_on_demand = false # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -71,8 +74,6 @@ tablename = f2b # protocol = tcp - - # Option: actiontype # Notes.: defines additions to the blocking rule # Values: leave empty to block all attempts from the host diff --git a/config/action.d/sendmail-buffered.conf b/config/action.d/sendmail-buffered.conf index 80eb20a3..37bc642d 100644 --- a/config/action.d/sendmail-buffered.conf +++ b/config/action.d/sendmail-buffered.conf @@ -10,11 +10,14 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD # -actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n` +actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on <fq-hostname> From: <sendername> <<sender>> To: <dest>\n Hi,\n @@ -28,7 +31,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n` # Values: CMD # actionstop = if [ -f <tmpfile> ]; then - printf %%b "Subject: [Fail2Ban] <name>: summary from `uname -n` + printf %%b "Subject: [Fail2Ban] <name>: summary from <fq-hostname> From: <sendername> <<sender>> To: <dest>\n Hi,\n @@ -38,7 +41,7 @@ actionstop = if [ -f <tmpfile> ]; then Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest> rm <tmpfile> fi - printf %%b "Subject: [Fail2Ban] <name>: stopped on `uname -n` + printf %%b "Subject: [Fail2Ban] <name>: stopped on <fq-hostname> From: Fail2Ban <<sender>> To: <dest>\n Hi,\n @@ -61,7 +64,7 @@ actioncheck = actionban = printf %%b "`date`: <ip> (<failures> failures)\n" >> <tmpfile> LINE=$( wc -l <tmpfile> | awk '{ print $1 }' ) if [ $LINE -ge <lines> ]; then - printf %%b "Subject: [Fail2Ban] <name>: summary from `uname -n` + printf %%b "Subject: [Fail2Ban] <name>: summary from <fq-hostname> From: <sendername> <<sender>> To: <dest>\n Hi,\n diff --git a/config/action.d/sendmail-common.conf b/config/action.d/sendmail-common.conf index 1475dedb..46eca9ca 100644 --- a/config/action.d/sendmail-common.conf +++ b/config/action.d/sendmail-common.conf @@ -14,7 +14,7 @@ after = sendmail-common.local # Notes.: command executed once at the start of Fail2Ban. # Values: CMD # -actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n` +actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n @@ -27,7 +27,7 @@ actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on `uname -n` # Notes.: command executed once at the end of Fail2Ban # Values: CMD # -actionstop = printf %%b "Subject: [Fail2Ban] <name>: stopped on `uname -n` +actionstop = printf %%b "Subject: [Fail2Ban] <name>: stopped on <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/sendmail-geoip-lines.conf b/config/action.d/sendmail-geoip-lines.conf index a5616e9f..b7c1bf36 100644 --- a/config/action.d/sendmail-geoip-lines.conf +++ b/config/action.d/sendmail-geoip-lines.conf @@ -11,6 +11,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: Command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. @@ -20,7 +23,7 @@ before = sendmail-common.conf # Tags: See jail.conf(5) man page # Values: CMD # -actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n @@ -33,7 +36,7 @@ actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n http://whois.domaintools.com/<ip>\n\n Country:`geoiplookup -f /usr/share/GeoIP/GeoIP.dat "<ip>" | cut -d':' -f2-` AS:`geoiplookup -f /usr/share/GeoIP/GeoIPASNum.dat "<ip>" | cut -d':' -f2-` - hostname: `host -t A <ip> 2>&1`\n\n + hostname: <ip-host>\n\n Lines containing failures of <ip>\n"; %(_grep_logs)s; printf %%b "\n diff --git a/config/action.d/sendmail-whois-ipjailmatches.conf b/config/action.d/sendmail-whois-ipjailmatches.conf index 689ffe45..06ea3a3e 100644 --- a/config/action.d/sendmail-whois-ipjailmatches.conf +++ b/config/action.d/sendmail-whois-ipjailmatches.conf @@ -10,13 +10,16 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/sendmail-whois-ipmatches.conf b/config/action.d/sendmail-whois-ipmatches.conf index b06e6db6..83bff1b4 100644 --- a/config/action.d/sendmail-whois-ipmatches.conf +++ b/config/action.d/sendmail-whois-ipmatches.conf @@ -10,13 +10,16 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/sendmail-whois-lines.conf b/config/action.d/sendmail-whois-lines.conf index e1c85928..4b947cb2 100644 --- a/config/action.d/sendmail-whois-lines.conf +++ b/config/action.d/sendmail-whois-lines.conf @@ -11,13 +11,16 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = ( printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/sendmail-whois-matches.conf b/config/action.d/sendmail-whois-matches.conf index 8bca5937..01520135 100644 --- a/config/action.d/sendmail-whois-matches.conf +++ b/config/action.d/sendmail-whois-matches.conf @@ -10,13 +10,16 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/sendmail-whois.conf b/config/action.d/sendmail-whois.conf index 55b80bc5..2fb01ed3 100644 --- a/config/action.d/sendmail-whois.conf +++ b/config/action.d/sendmail-whois.conf @@ -10,13 +10,16 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/sendmail.conf b/config/action.d/sendmail.conf index 5f5670c3..cf420915 100644 --- a/config/action.d/sendmail.conf +++ b/config/action.d/sendmail.conf @@ -10,13 +10,16 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from `uname -n` +actionban = printf %%b "Subject: [Fail2Ban] <name>: banned <ip> from <fq-hostname> Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <sendername> <<sender>> To: <dest>\n diff --git a/config/action.d/smtp.py b/config/action.d/smtp.py index 2429cf48..9cdfe327 100644 --- a/config/action.d/smtp.py +++ b/config/action.d/smtp.py @@ -123,9 +123,12 @@ class SMTPAction(ActionBase): self.message_values = CallingMap( jailname = self._jail.name, hostname = socket.gethostname, - bantime = self._jail.actions.getBanTime, + bantime = lambda: self._jail.actions.getBanTime(), ) + # bypass ban/unban for restored tickets + self.norestored = 1 + def _sendMessage(self, subject, text): """Sends message based on arguments and instance's properties. @@ -211,6 +214,8 @@ class SMTPAction(ActionBase): Dictionary which includes information in relation to the ban. """ + if aInfo.get('restored'): + return aInfo.update(self.message_values) message = "".join([ messages['ban']['head'], diff --git a/config/action.d/xarf-login-attack.conf b/config/action.d/xarf-login-attack.conf index 3ab73817..d041482a 100644 --- a/config/action.d/xarf-login-attack.conf +++ b/config/action.d/xarf-login-attack.conf @@ -22,7 +22,7 @@ # Login-Attack, Malware-Attack, Fraud (Phishing, etc.), Info DNSBL # # For details see: -# https://github.com/abusix/xarf-specification +# https://github.com/xarf/xarf-specification # http://www.x-arf.org/schemata.html # # Author: Daniel Black @@ -32,6 +32,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + actionstart = actionstop = @@ -43,7 +46,7 @@ actionban = oifs=${IFS}; IFS=.;SEP_IP=( <ip> ); set -- ${SEP_IP}; ADDRESSES=$(di FROM=<sender> SERVICE=<service> FAILURES=<failures> - REPORTID=<time>@`uname -n` + REPORTID=<time>@<fq-hostname> TLP=<tlp> PORT=<port> DATE=`LC_ALL=C date --date=@<time> +"%%a, %%d %%h %%Y %%T %%z"` @@ -116,7 +119,7 @@ logpath = /dev/null # Option: sender # Notes.: This is the sender that is included in the XARF report -sender = fail2ban@`uname -n` +sender = fail2ban@<fq-hostname> # Option: port # Notes.: This is the port number that received the login-attack diff --git a/config/filter.d/apache-auth.conf b/config/filter.d/apache-auth.conf index 8a63858d..d9a6fa5e 100644 --- a/config/filter.d/apache-auth.conf +++ b/config/filter.d/apache-auth.conf @@ -9,20 +9,21 @@ before = apache-common.conf [Definition] +prefregex = ^%(_apache_error_client)s (?:AH\d+: )?<F-CONTENT>.+</F-CONTENT>$ -failregex = ^%(_apache_error_client)s (AH(01797|01630): )?client denied by server configuration: (uri )?\S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01617: )?user .*? authentication failure for "\S*": Password Mismatch(, referer: \S+)?$ - ^%(_apache_error_client)s (AH01618: )?user .*? not found(: )?\S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01614: )?client used wrong authentication scheme: \S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH\d+: )?Authorization of user \S+ to access \S* failed, reason: .*$ - ^%(_apache_error_client)s (AH0179[24]: )?(Digest: )?user .*?: password mismatch: \S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH0179[01]: |Digest: )user `.*?' in realm `.+' (not found|denied by provider): \S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01631: )?user .*?: authorization failure for "\S*":(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01775: )?(Digest: )?invalid nonce .* received - length is not \S+(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01788: )?(Digest: )?realm mismatch - got `.*?' but expected `.+'(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01789: )?(Digest: )?unknown algorithm `.*?' received: \S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01793: )?invalid qop `.*?' received: \S*(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01777: )?(Digest: )?invalid nonce .*? received - user attempted time travel(, referer: \S+)?\s*$ +# auth_type = ((?:Digest|Basic): )? +auth_type = ([A-Z]\w+: )? + +failregex = ^client (?:denied by server configuration|used wrong authentication scheme)\b + ^user <F-USER>(?:\S*|.*?)</F-USER> (?:auth(?:oriz|entic)ation failure|not found|denied by provider)\b + ^Authorization of user <F-USER>(?:\S*|.*?)</F-USER> to access .*? failed\b + ^%(auth_type)suser <F-USER>(?:\S*|.*?)</F-USER>: password mismatch\b + ^%(auth_type)suser `<F-USER>(?:[^']*|.*?)</F-USER>' in realm `.+' (not found|denied by provider)\b + ^%(auth_type)sinvalid nonce .* received - length is not\b + ^%(auth_type)srealm mismatch - got `(?:[^']*|.*?)' but expected\b + ^%(auth_type)sunknown algorithm `(?:[^']*|.*?)' received\b + ^invalid qop `(?:[^']*|.*?)' received\b + ^%(auth_type)sinvalid nonce .*? received - user attempted time travel\b ignoreregex = @@ -43,14 +44,17 @@ ignoreregex = # all of these expressions. Lots of submodules like mod_authz_* return back to mod_authz_core # to return the actual failure. # +# Note that URI can contain spaces. +# # See also: http://wiki.apache.org/httpd/ListOfErrors # Expressions that don't have tests and aren't common. # more be added with https://issues.apache.org/bugzilla/show_bug.cgi?id=55284 -# ^%(_apache_error_client)s (AH01778: )?user .*: nonce expired \([\d.]+ seconds old - max lifetime [\d.]+\) - sending new nonce\s*$ -# ^%(_apache_error_client)s (AH01779: )?user .*: one-time-nonce mismatch - sending new nonce\s*$ -# ^%(_apache_error_client)s (AH02486: )?realm mismatch - got `.*' but no realm specified\s*$ +# ^user .*: nonce expired \([\d.]+ seconds old - max lifetime [\d.]+\) - sending new nonce\s*$ +# ^user .*: one-time-nonce mismatch - sending new nonce\s*$ +# ^realm mismatch - got `(?:[^']*|.*?)' but no realm specified\s*$ # -# referer is always in error log messages if it exists added as per the log_error_core function in server/log.c +# Because url/referer are foreign input, short form of regex used if long enough to idetify failure. # # Author: Cyril Jaquier -# Major edits by Daniel Black +# Major edits by Daniel Black and Ben Rubson. +# Rewritten for v.0.10 by Sergey Brester (sebres). diff --git a/config/filter.d/apache-botsearch.conf b/config/filter.d/apache-botsearch.conf index 5687d405..7def09ac 100644 --- a/config/filter.d/apache-botsearch.conf +++ b/config/filter.d/apache-botsearch.conf @@ -23,13 +23,12 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ((AH001(28|30): )?File does not exist|(AH01264: )?script not found or unable to stat): <webroot><block>(, referer: \S+)?\s*$ - ^%(_apache_error_client)s script '<webroot><block>' not found or unable to stat(, referer: \S+)?\s*$ - -ignoreregex = +prefregex = ^%(_apache_error_client)s (?:AH\d+: )?<F-CONTENT>.+</F-CONTENT>$ +failregex = ^(?:File does not exist|script not found or unable to stat): <webroot><block>(, referer: \S+)?\s*$ + ^script '<webroot><block>' not found or unable to stat(, referer: \S+)?\s*$ -[Init] +ignoreregex = # Webroot represents the webroot on which all other files are based webroot = /var/www/ diff --git a/config/filter.d/apache-common.conf b/config/filter.d/apache-common.conf index 3d1f902b..3eec83d0 100644 --- a/config/filter.d/apache-common.conf +++ b/config/filter.d/apache-common.conf @@ -3,12 +3,31 @@ [INCLUDES] +before = common.conf # Load customizations if any available after = apache-common.local [DEFAULT] -_apache_error_client = \[\] \[(:?error|\S+:\S+)\]( \[pid \d+(:\S+ \d+)?\])? \[client <HOST>(:\d{1,5})?\] +# Apache logging mode: +# all - universal prefix (logfile, syslog) +# logfile - logfile only +# syslog - syslog only +# Use `filter = apache-auth[logging=syslog]` to get more precise regex if apache logs into syslog (ErrorLog syslog). +# Use `filter = apache-auth[logging=all]` to get universal regex matches both logging variants. +logging = logfile + +# Apache logging prefixes (date-pattern prefix, server, process etc.): +apache-prefix-syslog = %(__prefix_line)s +apache-prefix-logfile = \[\]\s +apache-prefix-all = (?:%(apache-prefix-logfile)s|%(apache-prefix-syslog)s)? + +# Setting for __prefix_line (only `logging=syslog`): +_daemon = (?:apache\d*|httpd(?:/\w+)?) + +apache-prefix = <apache-prefix-<logging>> + +_apache_error_client = <apache-prefix>\[(:?error|\S+:\S+)\]( \[pid \d+(:\S+ \d+)?\])? \[client <HOST>(:\d{1,5})?\] datepattern = {^LN-BEG} diff --git a/config/filter.d/apache-modsecurity.conf b/config/filter.d/apache-modsecurity.conf index ad7e9b24..e296227a 100644 --- a/config/filter.d/apache-modsecurity.conf +++ b/config/filter.d/apache-modsecurity.conf @@ -10,9 +10,10 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ModSecurity: (\[.*?\] )*Access denied with code [45]\d\d.*$ +failregex = ^%(_apache_error_client)s ModSecurity:\s+(?:\[(?:\w+ \"[^\"]*\"|[^\]]*)\]\s*)*Access denied with code [45]\d\d ignoreregex = # https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats # Author: Daniel Black +# Sergey G. Brester aka sebres (review, optimization) diff --git a/config/filter.d/apache-overflows.conf b/config/filter.d/apache-overflows.conf index 74e44b8e..02a2ef20 100644 --- a/config/filter.d/apache-overflows.conf +++ b/config/filter.d/apache-overflows.conf @@ -8,11 +8,15 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ((AH0013[456]: )?Invalid (method|URI) in request .*( - possible attempt to establish SSL connection on non-SSL port)?|(AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string: .*|AH00566: request failed: invalid characters in URI)(, referer: \S+)?$ +failregex = ^%(_apache_error_client)s (?:(?:AH0013[456]: )?Invalid (method|URI) in request\b|(?:AH00565: )?request failed: URI too long \(longer than \d+\)|request failed: erroneous characters after protocol string:|(?:AH00566: )?request failed: invalid characters in URI\b) ignoreregex = # DEV Notes: +# +# [sebres] Because this apache-log could contain very long URLs (and/or referrer), +# the parsing of it anchored way may be very vulnerable (at least as regards +# the system resources, see gh-1790). Thus rewritten without end-anchor ($). # # fgrep -r 'URI too long' httpd-2.* # httpd-2.2.25/server/protocol.c: "request failed: URI too long (longer than %d)", r->server->limit_req_line); diff --git a/config/filter.d/apache-shellshock.conf b/config/filter.d/apache-shellshock.conf index 39df1704..e2707dc0 100644 --- a/config/filter.d/apache-shellshock.conf +++ b/config/filter.d/apache-shellshock.conf @@ -9,8 +9,10 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s (AH01215: )?/bin/(ba)?sh: warning: HTTP_.*?: ignoring function definition attempt(, referer: \S+)?\s*$ - ^%(_apache_error_client)s (AH01215: )?/bin/(ba)?sh: error importing function definition for `HTTP_.*?'(, referer: \S+)?\s*$ +prefregex = ^%(_apache_error_client)s (AH01215: )?/bin/([bd]a)?sh: <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^warning: HTTP_[^:]+: ignoring function definition attempt(, referer: \S+)?\s*$ + ^error importing function definition for `HTTP_[^']+'(, referer: \S+)?\s*$ ignoreregex = diff --git a/config/filter.d/assp.conf b/config/filter.d/assp.conf index 7cca81d2..9837f71d 100644 --- a/config/filter.d/assp.conf +++ b/config/filter.d/assp.conf @@ -8,7 +8,7 @@ # [Definition] -# Note: First three failregex matches below are for ASSP V1 with the remaining being designed for V2. Deleting the V1 regex is recommended but I left it in for compatibilty reasons. +# Note: First three failregex matches below are for ASSP V1 with the remaining being designed for V2. Deleting the V1 regex is recommended but I left it in for compatibility reasons. __assp_actions = (?:dropping|refusing) diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index df55d288..f382e258 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -11,23 +11,27 @@ before = common.conf _daemon = asterisk -__pid_re = (?:\[\d+\]) +__pid_re = (?:\s*\[\d+\]) iso8601 = \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+[+-]\d{4} # All Asterisk log messages begin like this: log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*(?:(?: in)? \w+:)? -failregex = ^%(__prefix_line)s%(log_prefix)s Registration from '[^']*' failed for '<HOST>(:\d+)?' - (Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$ - ^%(__prefix_line)s%(log_prefix)s Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context - ^%(__prefix_line)s%(log_prefix)s Host <HOST> failed to authenticate as '[^']*'$ - ^%(__prefix_line)s%(log_prefix)s No registration for peer '[^']*' \(from <HOST>\)$ - ^%(__prefix_line)s%(log_prefix)s Host <HOST> failed MD5 authentication for '[^']*' \([^)]+\)$ - ^%(__prefix_line)s%(log_prefix)s Failed to authenticate (user|device) [^@]+@<HOST>\S*$ - ^%(__prefix_line)s%(log_prefix)s hacking attempt detected '<HOST>'$ - ^%(__prefix_line)s%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="([\d-]+|%(iso8601)s)",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="(\d*|<unknown>)",SessionID=".+",LocalAddress="IPV[46]/(UDP|TCP|WS)/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UDP|TCP|WS)/<HOST>/\d+"(,Challenge="[\w/]+")?(,ReceivedChallenge="\w+")?(,Response="\w+",ExpectedResponse="\w*")?(,ReceivedHash="[\da-f]+")?(,ACLName="\w+")?$ - ^%(__prefix_line)s%(log_prefix)s "Rejecting unknown SIP connection from <HOST>"$ - ^%(__prefix_line)s%(log_prefix)s Request (?:'[^']*' )?from '[^']*' failed for '<HOST>(?::\d+)?'\s\(callid: [^\)]*\) - (?:No matching endpoint found|Not match Endpoint(?: Contact)? ACL|(?:Failed|Error) to authenticate)\s*$ +prefregex = ^%(__prefix_line)s%(log_prefix)s <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^Registration from '[^']*' failed for '<HOST>(:\d+)?' - (?:Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$ + ^Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context + ^(?:Host )?<HOST> (?:failed (?:to authenticate\b|MD5 authentication\b)|tried to authenticate with nonexistent user\b) + ^No registration for peer '[^']*' \(from <HOST>\)$ + ^Failed to authenticate (?:user|device) [^@]+@<HOST>\S*$ + ^hacking attempt detected '<HOST>'$ + ^SecurityEvent="(?:FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)"(?:(?:,(?!RemoteAddress=)\w+="[^"]*")*|.*?),RemoteAddress="IPV[46]/(UDP|TCP|WS)/<HOST>/\d+"(?:,(?!RemoteAddress=)\w+="[^"]*")*$ + ^"Rejecting unknown SIP connection from <HOST>"$ + ^Request (?:'[^']*' )?from '[^']*' failed for '<HOST>(?::\d+)?'\s\(callid: [^\)]*\) - (?:No matching endpoint found|Not match Endpoint(?: Contact)? ACL|(?:Failed|Error) to authenticate)\s*$ + +# FreePBX (todo: make optional in v.0.10): +# ^(%(__prefix_line)s|\[\]\s*WARNING%(__pid_re)s:?(?:\[C-[\da-f]*\])? )[^:]+: Friendly Scanner from <HOST>$ ignoreregex = diff --git a/config/filter.d/botsearch-common.conf b/config/filter.d/botsearch-common.conf index a673a310..6052fab1 100644 --- a/config/filter.d/botsearch-common.conf +++ b/config/filter.d/botsearch-common.conf @@ -5,7 +5,7 @@ # Block is the actual non-found directories to block block = \/?(<webmail>|<phpmyadmin>|<wordpress>|cgi-bin|mysqladmin)[^,]* -# These are just convient definitions that assist the blocking of stuff that +# These are just convenient definitions that assist the blocking of stuff that # isn't installed webmail = roundcube|(ext)?mail|horde|(v-?)?webmail diff --git a/config/filter.d/courier-auth.conf b/config/filter.d/courier-auth.conf index 87ee55b4..1ac33736 100644 --- a/config/filter.d/courier-auth.conf +++ b/config/filter.d/courier-auth.conf @@ -11,7 +11,7 @@ before = common.conf _daemon = (?:courier)?(?:imapd?|pop3d?)(?:login)?(?:-ssl)? -failregex = ^%(__prefix_line)sLOGIN FAILED, user=.*, ip=\[<HOST>\]$ +failregex = ^%(__prefix_line)sLOGIN FAILED, (?:user|method)=.*, ip=\[<HOST>\]$ ignoreregex = diff --git a/config/filter.d/courier-smtp.conf b/config/filter.d/courier-smtp.conf index fc0afc26..888753c4 100644 --- a/config/filter.d/courier-smtp.conf +++ b/config/filter.d/courier-smtp.conf @@ -12,8 +12,10 @@ before = common.conf _daemon = courieresmtpd -failregex = ^%(__prefix_line)serror,relay=<HOST>,.*: 550 User (<.*> )?unknown\.?$ - ^%(__prefix_line)serror,relay=<HOST>,msg="535 Authentication failed\.",cmd:( AUTH \S+)?( [0-9a-zA-Z\+/=]+)?(?: \S+)$ +prefregex = ^%(__prefix_line)serror,relay=<HOST>,<F-CONTENT>.+</F-CONTENT>$ + +failregex = ^[^:]*: 550 User (<.*> )?unknown\.?$ + ^msg="535 Authentication failed\.",cmd:( AUTH \S+)?( [0-9a-zA-Z\+/=]+)?(?: \S+)$ ignoreregex = diff --git a/config/filter.d/cyrus-imap.conf b/config/filter.d/cyrus-imap.conf index 73764d9d..31dfda60 100644 --- a/config/filter.d/cyrus-imap.conf +++ b/config/filter.d/cyrus-imap.conf @@ -13,7 +13,7 @@ before = common.conf _daemon = (?:cyrus/)?(?:imap(d|s)?|pop3(d|s)?) -failregex = ^%(__prefix_line)sbadlogin: \S+ ?\[<HOST>\] \S+ .*?\[?SASL\(-13\): (authentication failure|user not found): .*\]?$ +failregex = ^%(__prefix_line)sbadlogin: [^\[]*\[<HOST>\] \S+ .*?\[?SASL\(-13\): (authentication failure|user not found): .*\]?$ ignoreregex = diff --git a/config/filter.d/domino-smtp.conf b/config/filter.d/domino-smtp.conf new file mode 100644 index 00000000..cdc17736 --- /dev/null +++ b/config/filter.d/domino-smtp.conf @@ -0,0 +1,47 @@ +# Fail2Ban configuration file for IBM Domino SMTP Server TASK to detect failed login attempts +# +# Author: Christian Brandlehner +# +# $Revision: 003 $ +# +# Configuration: +# Set the following Domino Server parameters in notes.ini: +# console_log_enabled=1 +# log_sessions=2 +# You also have to use a date and time format supported by fail2ban. Recommended notes.ini configuration is: +# DateOrder=DMY +# DateSeparator=- +# ClockType=24_Hour +# TimeSeparator=: +# +# Depending on your locale you might have to tweak the date and time format so fail2ban can read the log + +#[INCLUDES] +# Read common prefixes. If any customizations available -- read them from +# common.local +#before = common.conf + +[Definition] +# Option: failregex +# Notes.: regex to match the password failure messages in the logfile. The +# host must be matched by a group named "host". The tag "<HOST>" can +# be used for standard IP/hostname matching and is only an alias for +# (?:::f{4,6}:)?(?P<host>\S+) +# Values: TEXT +# +# Sample log entries (used different time formats and an extra sample with process info in front of date) +# 01-23-2009 19:54:51 SMTP Server: Authentication failed for user postmaster ; connecting host 1.2.3.4 +# [28325:00010-3735542592] 22-06-2014 09:56:12 smtp: postmaster [1.2.3.4] authentication failure using internet password +# 08-09-2014 06:14:27 smtp: postmaster [1.2.3.4] authentication failure using internet password +# 08-09-2014 06:14:27 SMTP Server: Authentication failed for user postmaster ; connecting host 1.2.3.4 + +__prefix = (?:\[[^\]]+\])?\s+ +failregex = ^%(__prefix)sSMTP Server: Authentication failed for user .*? \; connecting host <HOST>$ + ^%(__prefix)ssmtp: (?:[^\[]+ )*\[<HOST>\] authentication failure using internet password\s*$ +# Option: ignoreregex +# Notes.: regex to ignore. If this regex matches, the line is ignored. +# Values: TEXT +# + +ignoreregex = + diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 6f8510fc..42ab2c88 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -7,13 +7,15 @@ before = common.conf [Definition] -_daemon = (auth|dovecot(-auth)?|auth-worker) +_auth_worker = (?:dovecot: )?auth(?:-worker)? +_daemon = (dovecot(-auth)?|auth) -failregex = ^%(__prefix_line)s(%(__pam_auth)s(\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(\s+user=\S*)?\s*$ - ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>(, lip=(\d{1,3}\.){3}\d{1,3})?(, TLS( handshaking(: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$ - ^%(__prefix_line)s(Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,<HOST>\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$ - ^%(__prefix_line)s(auth|auth-worker\(\d+\)): (pam|passwd-file)\(\S+,<HOST>\): unknown user\s*$ - ^%(__prefix_line)s(auth|auth-worker\(\d+\)): Info: ldap\(\S*,<HOST>,\S*\): invalid credentials\s*$ +prefregex = ^%(__prefix_line)s(%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap)-login: )?(?:Info: )?<F-CONTENT>.+</F-CONTENT>$ + +failregex = ^authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=<HOST>(?:\s+user=\S*)?\s*$ + ^(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<[^>]*>,)?( method=\S+,)? rip=<HOST>(?:, lip=\S+)?(?:, TLS(?: handshaking(?:: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$ + ^pam\(\S+,<HOST>\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$ + ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:unknown user|invalid credentials)\s*$ ignoreregex = @@ -31,3 +33,4 @@ datepattern = {^LN-BEG}TAI64N # Author: Martin Waschbuesch # Daniel Black (rewrote with begin and end anchors) # Martin O'Neal (added LDAP authentication failure regex) +# Sergey G. Brester aka sebres (reviewed, optimized, IPv6-compatibility) diff --git a/config/filter.d/dropbear.conf b/config/filter.d/dropbear.conf index 288b0882..930bb128 100644 --- a/config/filter.d/dropbear.conf +++ b/config/filter.d/dropbear.conf @@ -23,9 +23,11 @@ before = common.conf _daemon = dropbear -failregex = ^%(__prefix_line)s[Ll]ogin attempt for nonexistent user ('.*' )?from <HOST>:\d+$ - ^%(__prefix_line)s[Bb]ad (PAM )?password attempt for .+ from <HOST>(:\d+)?$ - ^%(__prefix_line)s[Ee]xit before auth \(user '.+', \d+ fails\): Max auth tries reached - user '.+' from <HOST>:\d+\s*$ +prefregex = ^%(__prefix_line)s<F-CONTENT>(?:[Ll]ogin|[Bb]ad|[Ee]xit).+</F-CONTENT>$ + +failregex = ^[Ll]ogin attempt for nonexistent user ('.*' )?from <HOST>:\d+$ + ^[Bb]ad (PAM )?password attempt for .+ from <HOST>(:\d+)?$ + ^[Ee]xit before auth \(user '.+', \d+ fails\): Max auth tries reached - user '.+' from <HOST>:\d+\s*$ ignoreregex = diff --git a/config/filter.d/ejabberd-auth.conf b/config/filter.d/ejabberd-auth.conf index 56517489..48e82df5 100644 --- a/config/filter.d/ejabberd-auth.conf +++ b/config/filter.d/ejabberd-auth.conf @@ -16,8 +16,8 @@ # searched for other failures. This tag can be used multiple times. # Values: TEXT # -failregex = ^=INFO REPORT==== ===\nI\(<0\.\d+\.0>:ejabberd_c2s:\d+\) : \([^)]+\) Failed authentication for .+ from IP <HOST> \({{(?:\d+,){3}\d+},\d+}\)$ - ^(?:\.\d+)? \[info\] <0\.\d+\.\d>@ejabberd_c2s:wait_for_feature_request:\d+ \([^\)]+\) Failed authentication for \S+ from IP <HOST>$ +failregex = ^=INFO REPORT==== ===\nI\(<0\.\d+\.0>:ejabberd_c2s:\d+\) : \([^)]+\) Failed authentication for \S+ from (?:IP )?<HOST>(?: \({{(?:\d+,){3}\d+},\d+}\))?$ + ^(?:\.\d+)? \[info\] <0\.\d+\.\d>@ejabberd_c2s:\w+:\d+ \([^\)]+\) Failed (?:c2s \w+ )?authentication for \S+ from (?:IP )?(?:::FFFF:)?<HOST>(?:: |$) # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/config/filter.d/exim-common.conf b/config/filter.d/exim-common.conf index 0e1b74fa..b3b25750 100644 --- a/config/filter.d/exim-common.conf +++ b/config/filter.d/exim-common.conf @@ -9,7 +9,9 @@ after = exim-common.local [Definition] -host_info = (?:H=([\w.-]+ )?(?:\(\S+\) )?)?\[<HOST>\](?::\d+)? (?:I=\[\S+\](:\d+)? )?(?:U=\S+ )?(?:P=e?smtp )? +host_info_pre = (?:H=([\w.-]+ )?(?:\(\S+\) )?)? +host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtp)?(?: F=(?:<>|[^@]+@\S+))?\s +host_info = %(host_info_pre)s\[<HOST>\]%(host_info_suf)s pid = (?: \[\d+\])? # DEV Notes: diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf index a1d699c0..2151a161 100644 --- a/config/filter.d/exim.conf +++ b/config/filter.d/exim.conf @@ -13,14 +13,17 @@ before = exim-common.conf [Definition] +# Fre-filter via "prefregex" is currently inactive because of too different failure syntax in exim-log (testing needed): +#prefregex = ^%(pid)s <F-CONTENT>\b(?:\w+ authenticator failed|([\w\-]+ )?SMTP (?:(?:call|connection) from|protocol(?: synchronization)? error)|no MAIL in|(?:%(host_info_pre)s\[[^\]]+\]%(host_info_suf)s(?:sender verify fail|rejected RCPT|dropped|AUTH command))).+</F-CONTENT>$ + failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$ - ^%(pid)s \w+ authenticator failed for (\S+ )?\(\S+\) \[<HOST>\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ - ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user)\s*$ + ^%(pid)s \w+ authenticator failed for (?:[^\[\( ]* )?(?:\(\S*\) )?\[<HOST>\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$ + ^%(pid)s %(host_info)srejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$ ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$ ^%(pid)s SMTP call from \S+ %(host_info)sdropped: too many nonmail commands \(last was "\S+"\)\s*$ ^%(pid)s SMTP protocol error in "AUTH \S*(?: \S*)?" %(host_info)sAUTH command used when not advertised\s*$ - ^%(pid)s no MAIL in SMTP connection from (?:\S* )?(?:\(\S*\) )?%(host_info)sD=\d+s(?: C=\S*)?\s*$ - ^%(pid)s \S+ SMTP connection from (?:\S* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$ + ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S+s(?: C=\S*)?\s*$ + ^%(pid)s (?:[\w\-]+ )?SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$ ignoreregex = diff --git a/config/filter.d/froxlor-auth.conf b/config/filter.d/froxlor-auth.conf index 04003263..d8f3785c 100644 --- a/config/filter.d/froxlor-auth.conf +++ b/config/filter.d/froxlor-auth.conf @@ -25,8 +25,11 @@ _daemon = Froxlor # (?:::f{4,6}:)?(?P<host>[\w\-.^_]+) # Values: TEXT # -failregex = ^%(__prefix_line)s\[Login Action <HOST>\] Unknown user \S* tried to login.$ - ^%(__prefix_line)s\[Login Action <HOST>\] User \S* tried to login with wrong password.$ + +prefregex = ^%(__prefix_line)s\[Login Action <HOST>\] <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^Unknown user \S* tried to login.$ + ^User \S* tried to login with wrong password.$ # Option: ignoreregex diff --git a/config/filter.d/haproxy-http-auth.conf b/config/filter.d/haproxy-http-auth.conf index 298ca292..f92f9d67 100644 --- a/config/filter.d/haproxy-http-auth.conf +++ b/config/filter.d/haproxy-http-auth.conf @@ -28,7 +28,7 @@ _daemon = haproxy # (?:::f{4,6}:)?(?P<host>[\w\-.^_]+) # Values: TEXT # -failregex = ^%(__prefix_line)s<HOST>.*<NOSRV> -1/-1/-1/-1/\+*\d* 401 +failregex = ^%(__prefix_line)s<HOST>(?::\d+)?\s+.*<NOSRV> -1/-1/-1/-1/\+*\d* 401 # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. diff --git a/config/filter.d/kerio.conf b/config/filter.d/kerio.conf index e0d94753..0fde0927 100644 --- a/config/filter.d/kerio.conf +++ b/config/filter.d/kerio.conf @@ -3,9 +3,14 @@ [Definition] failregex = ^ SMTP Spam attack detected from <HOST>, - ^ IP address <HOST> found in DNS blacklist \S+, mail from \S+ to \S+$ + ^ IP address <HOST> found in DNS blacklist ^ Relay attempt from IP address <HOST> ^ Attempt to deliver to unknown recipient \S+, from \S+, IP address <HOST>$ + ^ Failed SMTP login from <HOST> + ^ SMTP: User \S+ doesn't exist. Attempt from IP address <HOST> + ^ Client with IP address <HOST> has no reverse DNS entry, connection rejected before SMTP greeting$ + ^ Administration login into Web Administration from <HOST> failed: IP address not allowed$ + ^ Message from IP address <HOST>, sender \S+ rejected: sender domain does not exist$ ignoreregex = @@ -14,5 +19,6 @@ datepattern = ^\[%%d/%%b/%%Y %%H:%%M:%%S\] # DEV NOTES: # # Author: A.P. Lawrence +# Updated by: M. Bischoff <https://github.com/herrbischoff> # # Based off: http://aplawrence.com/Kerio/fail2ban.html diff --git a/config/filter.d/mongodb-auth.conf b/config/filter.d/mongodb-auth.conf new file mode 100644 index 00000000..66c27abb --- /dev/null +++ b/config/filter.d/mongodb-auth.conf @@ -0,0 +1,49 @@ +# Fail2Ban filter for unsuccesfull MongoDB authentication attempts +# +# Logfile /var/log/mongodb/mongodb.log +# +# add setting in /etc/mongodb.conf +# logpath=/var/log/mongodb/mongodb.log +# +# and use of the authentication +# auth = true +# + +[Definition] +#failregex = ^\s+\[initandlisten\] connection accepted from <HOST>:\d+ \#(?P<__connid>\d+) \(1 connection now open\)<SKIPLINES>\s+\[conn(?P=__connid)\] Failed to authenticate\s+ +failregex = ^\s+\[conn(?P<__connid>\d+)\] Failed to authenticate [^\n]+<SKIPLINES>\s+\[conn(?P=__connid)\] end connection <HOST> + +ignoreregex = + + +[Init] +maxlines = 10 + +# DEV Notes: +# +# Regarding the multiline regex: +# +# There can be a nunber of non-related lines between the first and second part +# of this regex maxlines of 10 is quite generious. +# +# Note the capture __connid, includes the connection ID, used in second part of regex. +# +# The first regex is commented out (but will match also), because it is better to use +# the host from "end connection" line (uncommented above): +# - it has the same prefix, searching begins directly with failure message +# (so faster, because ignores success connections at all) +# - it is not so vulnerable in case of possible race condition +# +# Log example: +# 2016-10-20T09:54:27.108+0200 [initandlisten] connection accepted from 127.0.0.1:53276 #1 (1 connection now open) +# 2016-10-20T09:54:27.109+0200 [conn1] authenticate db: test { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# 2016-10-20T09:54:27.110+0200 [conn1] Failed to authenticate root@test with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@test +# 2016-11-09T09:54:27.894+0100 [conn1] end connection 127.0.0.1:53276 (0 connections now open) +# 2016-11-09T11:55:58.890+0100 [initandlisten] connection accepted from 127.0.0.1:54266 #1510 (1 connection now open) +# 2016-11-09T11:55:58.892+0100 [conn1510] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# 2016-11-09T11:55:58.892+0100 [conn1510] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# 2016-11-09T11:55:58.894+0100 [conn1510] end connection 127.0.0.1:54266 (0 connections now open) +# +# Authors: Alexander Finkhäuser +# Sergey G. Brester (sebres) + diff --git a/config/filter.d/murmur.conf b/config/filter.d/murmur.conf index 507bbd2f..f5f100a6 100644 --- a/config/filter.d/murmur.conf +++ b/config/filter.d/murmur.conf @@ -17,8 +17,10 @@ _usernameregex = [^>]+ _prefix = \s+\d+ => <\d+:%(_usernameregex)s\(-1\)> Rejected connection from <HOST>:\d+: -failregex = ^%(_prefix)s Invalid server password$ - ^%(_prefix)s Wrong certificate or password for existing user$ +prefregex = ^%(_prefix)s <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^Invalid server password$ + ^Wrong certificate or password for existing user$ ignoreregex = diff --git a/config/filter.d/mysqld-auth.conf b/config/filter.d/mysqld-auth.conf index 3ad70cb7..31bd2056 100644 --- a/config/filter.d/mysqld-auth.conf +++ b/config/filter.d/mysqld-auth.conf @@ -1,4 +1,4 @@ -# Fail2Ban filter for unsuccesfull MySQL authentication attempts +# Fail2Ban filter for unsuccesful MySQL authentication attempts # # # To log wrong MySQL access attempts add to /etc/my.cnf in [mysqld]: diff --git a/config/filter.d/named-refused.conf b/config/filter.d/named-refused.conf index eec3d667..2e14d442 100644 --- a/config/filter.d/named-refused.conf +++ b/config/filter.d/named-refused.conf @@ -34,9 +34,11 @@ __daemon_combs_re=(?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re) # this can be optional (for instance if we match named native log files) __line_prefix=(?:\s\S+ %(__daemon_combs_re)s\s+)? -failregex = ^%(__line_prefix)s( error:)?\s*client <HOST>#\S+( \([\S.]+\))?: (view (internal|external): )?query(?: \(cache\))? '.*' denied\s*$ - ^%(__line_prefix)s( error:)?\s*client <HOST>#\S+( \([\S.]+\))?: zone transfer '\S+/AXFR/\w+' denied\s*$ - ^%(__line_prefix)s( error:)?\s*client <HOST>#\S+( \([\S.]+\))?: bad zone transfer request: '\S+/IN': non-authoritative zone \(NOTAUTH\)\s*$ +prefregex = ^%(__line_prefix)s( error:)?\s*client <HOST>#\S+( \([\S.]+\))?: <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^(view (internal|external): )?query(?: \(cache\))? '.*' denied\s*$ + ^zone transfer '\S+/AXFR/\w+' denied\s*$ + ^bad zone transfer request: '\S+/IN': non-authoritative zone \(NOTAUTH\)\s*$ ignoreregex = diff --git a/config/filter.d/pam-generic.conf b/config/filter.d/pam-generic.conf index e0d4e9c1..ff4ea802 100644 --- a/config/filter.d/pam-generic.conf +++ b/config/filter.d/pam-generic.conf @@ -16,7 +16,12 @@ _ttys_re=\S* __pam_re=\(?%(__pam_auth)s(?:\(\S+\))?\)?:? _daemon = \S+ -failregex = ^%(__prefix_line)s%(__pam_re)s\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=%(_ttys_re)s ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$ +prefregex = ^%(__prefix_line)s%(__pam_re)s\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=%(_ttys_re)s <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^ruser=<F-USER>\S*</F-USER> rhost=<HOST>\s*$ + ^ruser= rhost=<HOST>\s+user=<F-USER>\S*</F-USER>\s*$ + ^ruser= rhost=<HOST>\s+user=<F-USER>.*?</F-USER>\s*$ + ^ruser=<F-USER>.*?</F-USER> rhost=<HOST>\s*$ ignoreregex = diff --git a/config/filter.d/postfix-rbl.conf b/config/filter.d/postfix-rbl.conf deleted file mode 100644 index c3f8c332..00000000 --- a/config/filter.d/postfix-rbl.conf +++ /dev/null @@ -1,19 +0,0 @@ -# Fail2Ban filter for Postfix's RBL based Blocked hosts -# -# - -[INCLUDES] - -# Read common prefixes. If any customizations available -- read them from -# common.local -before = common.conf - -[Definition] - -_daemon = postfix(-\w+)?/smtpd - -failregex = ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 454 4\.7\.1 Service unavailable; Client host \[\S+\] blocked using .* from=<\S*> to=<\S+> proto=ESMTP helo=<\S*>$ - -ignoreregex = - -# Author: Lee Clemens diff --git a/config/filter.d/postfix-sasl.conf b/config/filter.d/postfix-sasl.conf deleted file mode 100644 index 1a24ca94..00000000 --- a/config/filter.d/postfix-sasl.conf +++ /dev/null @@ -1,21 +0,0 @@ -# Fail2Ban filter for postfix authentication failures -# - -[INCLUDES] - -before = common.conf - -[Definition] - -_daemon = postfix(-\w+)?/(?:submission/|smtps/)?smtp[ds] - -failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(:[ A-Za-z0-9+/:]*={0,2})?\s*$ - -ignoreregex = authentication failed: Connection lost to authentication server$ - -[Init] - -journalmatch = _SYSTEMD_UNIT=postfix.service - - -# Author: Yaroslav Halchenko diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index 3051409b..b86b3d4d 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -10,15 +10,57 @@ before = common.conf [Definition] -_daemon = postfix(-\w+)?/(?:submission/|smtps/)?smtp[ds] - -failregex = ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 554 5\.7\.1 .*$ - ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 450 4\.7\.1 Client host rejected: cannot find your hostname, (\[\S*\]); from=<\S*> to=<\S+> proto=ESMTP helo=<\S*>$ - ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 450 4\.7\.1 : Helo command rejected: Host not found; from=<> to=<> proto=ESMTP helo= *$ - ^%(__prefix_line)sNOQUEUE: reject: EHLO from \S+\[<HOST>\]: 504 5\.5\.2 <\S+>: Helo command rejected: need fully-qualified hostname; - ^%(__prefix_line)sNOQUEUE: reject: VRFY from \S+\[<HOST>\]: 550 5\.1\.1 .*$ - ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 450 4\.1\.8 <\S*>: Sender address rejected: Domain not found; from=<\S*> to=<\S+> proto=ESMTP helo=<\S*>$ - ^%(__prefix_line)simproper command pipelining after \S+ from [^[]*\[<HOST>\]:?$ +_daemon = postfix(-\w+)?/\w+(?:/smtp[ds])? +_port = (?::\d+)? + +prefregex = ^%(__prefix_line)s<mdpr-<mode>> <F-CONTENT>.+</F-CONTENT>$ + +mdpr-normal = (?:NOQUEUE: reject:|improper command pipelining after \S+) +mdre-normal=^RCPT from [^[]*\[<HOST>\]<_port>: 55[04] 5\.7\.1\s + ^RCPT from [^[]*\[<HOST>\]<_port>: 45[04] 4\.7\.1 (?:Service unavailable\b|Client host rejected: cannot find your (reverse )?hostname\b) + ^RCPT from [^[]*\[<HOST>\]<_port>: 450 4\.7\.1 (<[^>]*>)?: Helo command rejected: Host not found\b + ^EHLO from [^[]*\[<HOST>\]<_port>: 504 5\.5\.2 (<[^>]*>)?: Helo command rejected: need fully-qualified hostname\b + ^VRFY from [^[]*\[<HOST>\]<_port>: 550 5\.1\.1\s + ^RCPT from [^[]*\[<HOST>\]<_port>: 450 4\.1\.8 (<[^>]*>)?: Sender address rejected: Domain not found\b + ^from [^[]*\[<HOST>\]:? + +mdpr-auth = warning: +mdre-auth = ^[^[]*\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed:(?! Connection lost to authentication server| Invalid authentication mechanism) +mdre-auth2= ^[^[]*\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed:(?! Connection lost to authentication server) +# todo: check/remove "Invalid authentication mechanism" from ignore list, if gh-1243 will get finished (see gh-1297). + +# Mode "rbl" currently included in mode "normal", but if needed for jail "postfix-rbl" only: +mdpr-rbl = %(mdpr-normal)s +mdre-rbl = ^RCPT from [^[]*\[<HOST>\]: [45]54 [45]\.7\.1 Service unavailable; Client host \[\S+\] blocked\b + +# Mode "rbl" currently included in mode "normal" (within 1st rule) +mdpr-more = %(mdpr-normal)s +mdre-more = %(mdre-normal)s + +mdpr-ddos = lost connection after(?! DATA) [A-Z]+ +mdre-ddos = ^from [^[]*\[<HOST>\]:? + +mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s) +mdre-extra = %(mdre-auth)s + %(mdre-normal)s + +mdpr-aggressive = (?:%(mdpr-auth)s|%(mdpr-normal)s|%(mdpr-ddos)s) +mdre-aggressive = %(mdre-auth2)s + %(mdre-normal)s + + + +failregex = <mdre-<mode>> + +# Parameter "mode": more (default combines normal and rbl), auth, normal, rbl, ddos, extra or aggressive (combines all) +# Usage example (for jail.local): +# [postfix] +# mode = aggressive +# # or another jail (rewrite filter parameters of jail): +# [postfix-rbl] +# filter = postfix[mode=rbl] +# +mode = more ignoreregex = diff --git a/config/filter.d/proftpd.conf b/config/filter.d/proftpd.conf index 4bc0ba01..a7bd2837 100644 --- a/config/filter.d/proftpd.conf +++ b/config/filter.d/proftpd.conf @@ -16,12 +16,19 @@ _daemon = proftpd __suffix_failed_login = (User not authorized for login|No such user found|Incorrect password|Password expired|Account disabled|Invalid shell: '\S+'|User in \S+|Limit (access|configuration) denies login|Not a UserAlias|maximum login length exceeded).? -failregex = ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ USER .*: no such user found from \S+ \[\S+\] to \S+:\S+ *$ - ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ USER .* \(Login failed\): %(__suffix_failed_login)s\s*$ - ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ SECURITY VIOLATION: .* login attempted\. *$ - ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ Maximum login attempts \(\d+\) exceeded *$ + +prefregex = ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ <F-CONTENT>(?:USER|SECURITY|Maximum).+</F-CONTENT>$ + + +failregex = ^USER .*: no such user found from \S+ \[\S+\] to \S+:\S+ *$ + ^USER .* \(Login failed\): %(__suffix_failed_login)s\s*$ + ^SECURITY VIOLATION: .* login attempted\. *$ + ^Maximum login attempts \(\d+\) exceeded *$ ignoreregex = +[Init] +journalmatch = _SYSTEMD_UNIT=proftpd.service + # Author: Yaroslav Halchenko # Daniel Black - hardening of regex diff --git a/config/filter.d/roundcube-auth.conf b/config/filter.d/roundcube-auth.conf index 886cf2d6..9912ff47 100644 --- a/config/filter.d/roundcube-auth.conf +++ b/config/filter.d/roundcube-auth.conf @@ -13,10 +13,15 @@ before = common.conf [Definition] -failregex = ^\s*(\[\])?(%(__hostname)s\s*(roundcube:)?\s*(<[\w]+>)? IMAP Error)?: (FAILED login|Login failed) for .*? from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$ - ^\[\]:\s*(<[\w]+>)? Failed login for [\w\-\.\+]+(@[\w\-\.\+]+\.[a-zA-Z]{2,6})? from <HOST> in session \w+( \(error: \d\))?$ +prefregex = ^\s*(\[\])?(%(__hostname)s\s*(?:roundcube(?:\[(\d*)\])?:)?\s*(<[\w]+>)? IMAP Error)?: <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^(?:FAILED login|Login failed) for <F-USER>.*</F-USER> from <HOST>(?:(?:\([^\)]*\))?\. (?:(?! from ).)*(?: user=(?P=user))? in \S+\.php on line \d+ \(\S+ \S+\))?$ + ^(?:<[\w]+> )?Failed login for <F-USER>.*</F-USER> from <HOST> in session \w+( \(error: \d\))?$ ignoreregex = + +journalmatch = SYSLOG_IDENTIFIER=roundcube + # DEV Notes: # # Source: https://github.com/roundcube/roundcubemail/blob/master/program/lib/Roundcube/rcube_imap.php#L180 diff --git a/config/filter.d/sendmail-reject.conf b/config/filter.d/sendmail-reject.conf index 20d3648e..0793a99b 100644 --- a/config/filter.d/sendmail-reject.conf +++ b/config/filter.d/sendmail-reject.conf @@ -21,30 +21,45 @@ before = common.conf _daemon = (?:(sm-(mta|acceptingconnections)|sendmail)) -failregex = ^%(__prefix_line)s\w{14}: ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[<HOST>\]( \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ - ^%(__prefix_line)sruleset=check_relay, arg1=(?P<dom>\S+), arg2=<HOST>, relay=((?P=dom) )?\[(\d+\.){3}\d+\]( \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ - ^%(__prefix_line)s\w{14}: rejecting commands from (\S* )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$ - ^%(__prefix_line)s\w{14}: (\S+ )?\[<HOST>\]: ((?i)expn|vrfy) \S+ \[rejected\]$ - ^(?P<__prefix>%(__prefix_line)s\w+: )<[^@]+@[^>]+>\.\.\. No such user here<SKIPLINES>(?P=__prefix)from=<[^@]+@[^>]+>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[<HOST>\]$ +prefregex = ^<F-MLFID>%(__prefix_line)s(?:\w{14}: )?</F-MLFID><F-CONTENT>.+</F-CONTENT>$ +cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[<HOST>\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ + ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=<HOST>, relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ + ^rejecting commands from (\S* )?\[<HOST>\] due to pre-greeting traffic after \d+ seconds$ + ^(?:\S+ )?\[<HOST>\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$ + ^<[^@]+@[^>]+>\.\.\. No such user here$ + ^<F-NOFAIL>from=<[^@]+@[^>]+></F-NOFAIL>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[<HOST>\]$ -ignoreregex = +mdre-normal = +mdre-extra = ^(?:\S+ )?\[<HOST>\](?: \(may be forged\))? did not issue (?:[A-Z]{4}[/ ]?)+during connection to M(?:TA|SP)(?:-\w+)?$ -[Init] +mdre-aggressive = %(mdre-extra)s + +failregex = %(cmnfailre)s + <mdre-<mode>> + +# Parameter "mode": normal (default), extra or aggressive +# Usage example (for jail.local): +# [sendmail-reject] +# filter = sendmail-reject[mode=extra] +# +mode = normal + +ignoreregex = -# "maxlines" is number of log lines to buffer for multi-line regex searches -maxlines = 10 # DEV NOTES: # -# Regarding the last multiline regex: +# Regarding the multiline regex: # -# There can be a nunber of non-related lines between the first and second part -# of this regex maxlines of 10 is quite generious. Only one of the -# "No such user" lines needs to be matched before the line with the HOST. +# "No such user" lines generate a failure and needs to be matched together with +# another line with the HOST, therefore no-failure line was added as regex, that +# contains HOST (see line with tag <F-NOFAIL>). # -# Note the capture __prefix, includes both the __prefix_lines (which includes -# the sendmail PID), but also the \w+ which the the sendmail assigned mail ID. +# Note the capture <F-MLFID>, includes both the __prefix_lines (which includes +# the sendmail PID), but also the `\w{14}` which the the sendmail assigned +# mail ID (todo: check this is necessary, possible obsolete). # -# Author: Daniel Black and Fabian Wenk +# Author: Daniel Black, Fabian Wenk and Sergey Brester aka sebres. +# Rewritten using prefregex by Serg G. Brester. diff --git a/config/filter.d/sshd-ddos.conf b/config/filter.d/sshd-ddos.conf deleted file mode 100644 index 4f71c7f3..00000000 --- a/config/filter.d/sshd-ddos.conf +++ /dev/null @@ -1,29 +0,0 @@ -# Fail2Ban ssh filter for at attempted exploit -# -# The regex here also relates to a exploit: -# -# http://www.securityfocus.com/bid/17958/exploit -# The example code here shows the pushing of the exploit straight after -# reading the server version. This is where the client version string normally -# pushed. As such the server will read this unparsible information as -# "Did not receive identification string". - -[INCLUDES] - -# Read common prefixes. If any customizations available -- read them from -# common.local -before = common.conf - -[Definition] - -_daemon = sshd - -failregex = ^%(__prefix_line)sDid not receive identification string from <HOST>\s*$ - -ignoreregex = - -[Init] - -journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd - -# Author: Yaroslav Halchenko diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index a425de1f..95915fcc 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -14,32 +14,74 @@ # common.local before = common.conf -[Definition] +[DEFAULT] _daemon = sshd -failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*$ - ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from <HOST>\s*$ - ^%(__prefix_line)sFailed \S+ for (?P<cond_inv>invalid user )?(?P<user>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)) from <HOST>(?: port \d+)?(?: ssh\d*)?(?(cond_user):|(?:(?:(?! from ).)*)$) - ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$ - ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$ - ^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*$ - ^%(__prefix_line)sUser .+ from <HOST> not allowed because listed in DenyUsers\s*$ - ^%(__prefix_line)sUser .+ from <HOST> not allowed because not in any group\s*$ - ^%(__prefix_line)srefused connect from \S+ \(<HOST>\)\s*$ - ^%(__prefix_line)s(?:error: )?Received disconnect from <HOST>: 3: .*: Auth fail(?: \[preauth\])?$ - ^%(__prefix_line)sUser .+ from <HOST> not allowed because a group is listed in DenyGroups\s*$ - ^%(__prefix_line)sUser .+ from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*$ - ^(?P<__prefix>%(__prefix_line)s)User .+ not allowed because account is locked<SKIPLINES>(?P=__prefix)(?:error: )?Received disconnect from <HOST>: 11: .+ \[preauth\]$ - ^(?P<__prefix>%(__prefix_line)s)Disconnecting: Too many authentication failures for .+? \[preauth\]<SKIPLINES>(?P=__prefix)(?:error: )?Connection closed by <HOST> \[preauth\]$ - ^(?P<__prefix>%(__prefix_line)s)Connection from <HOST> port \d+(?: on \S+ port \d+)?<SKIPLINES>(?P=__prefix)Disconnecting: Too many authentication failures for .+? \[preauth\]$ - ^%(__prefix_line)s(error: )?maximum authentication attempts exceeded for .* from <HOST>(?: port \d*)?(?: ssh\d*)? \[preauth\]$ - ^%(__prefix_line)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=<HOST>\s.*$ +# optional prefix (logged from several ssh versions) like "error: ", "error: PAM: " or "fatal: " +__pref = (?:(?:error|fatal): (?:PAM: )?)? +# optional suffix (logged from several ssh versions) like " [preauth]" +__suff = (?: \[preauth\])?\s* +__on_port_opt = (?: port \d+)?(?: on \S+(?: port \d+)?)? + +[Definition] + +prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID>%(__pref)s<F-CONTENT>.+</F-CONTENT>$ + +cmnfailre = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \S+)?\s*%(__suff)s$ + ^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>\s*%(__suff)s$ + ^Failed \S+ for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$) + ^<F-USER>ROOT</F-USER> LOGIN REFUSED.* FROM <HOST>\s*%(__suff)s$ + ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__on_port_opt)s\s*$ + ^User <F-USER>.+</F-USER> from <HOST> not allowed because not listed in AllowUsers\s*%(__suff)s$ + ^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers\s*%(__suff)s$ + ^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group\s*%(__suff)s$ + ^refused connect from \S+ \(<HOST>\)\s*%(__suff)s$ + ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$ + ^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups\s*%(__suff)s$ + ^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$ + ^pam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=<F-USER>\S*</F-USER>\s*rhost=<HOST>\s.*%(__suff)s$ + ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?%(__suff)s$ + ^User <F-USER>.+</F-USER> not allowed because account is locked%(__suff)s + ^<F-MLFFORGET>Disconnecting</F-MLFFORGET>: Too many authentication failures(?: for <F-USER>.+?</F-USER>)?%(__suff)s + ^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>: 11: + ^<F-NOFAIL>Connection <F-MLFFORGET>closed</F-MLFFORGET></F-NOFAIL> by <HOST>%(__suff)s$ + +mdre-normal = + +mdre-ddos = ^Did not receive identification string from <HOST>%(__suff)s$ + ^Connection <F-MLFFORGET>reset</F-MLFFORGET> by <HOST>%(__on_port_opt)s%(__suff)s + ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+: + ^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer%(__suff)s + +mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$ + ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found. + ^Unable to negotiate a (?:cipher|key exchange method)%(__suff)s$ + +mdre-aggressive = %(mdre-ddos)s + %(mdre-extra)s + +cfooterre = ^<F-NOFAIL>Connection from</F-NOFAIL> <HOST> + +failregex = %(cmnfailre)s + <mdre-<mode>> + %(cfooterre)s + +# Parameter "mode": normal (default), ddos, extra or aggressive (combines all) +# Usage example (for jail.local): +# [sshd] +# mode = extra +# # or another jail (rewrite filter parameters of jail): +# [sshd-aggressive] +# filter = sshd[mode=aggressive] +# +mode = normal + +#filter = sshd[mode=aggressive] ignoreregex = -# "maxlines" is number of log lines to buffer for multi-line regex searches -maxlines = 10 +maxlines = 1 journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd @@ -52,5 +94,5 @@ datepattern = {^LN-BEG} # and later catch-all's could contain user-provided input, which need to be greedily # matched away first. # -# Author: Cyril Jaquier, Yaroslav Halchenko, Petr Voralek, Daniel Black - +# Author: Cyril Jaquier, Yaroslav Halchenko, Petr Voralek, Daniel Black and Sergey Brester aka sebres +# Rewritten using prefregex (and introduced "mode" parameter) by Serg G. Brester. diff --git a/config/filter.d/suhosin.conf b/config/filter.d/suhosin.conf index f125eadc..46fbe381 100644 --- a/config/filter.d/suhosin.conf +++ b/config/filter.d/suhosin.conf @@ -17,7 +17,7 @@ _daemon = (?:lighttpd|suhosin) _lighttpd_prefix = (?:\(mod_fastcgi\.c\.\d+\) FastCGI-stderr:\s) -failregex = ^%(__prefix_line)s%(_lighttpd_prefix)s?ALERT - .* \(attacker '<HOST>', file '.*'(?:, line \d+)?\)$ +failregex = ^%(__prefix_line)s%(_lighttpd_prefix)s?ALERT - .*? \(attacker '<HOST>', file '[^']*'(?:, line \d+)?\)$ ignoreregex = diff --git a/config/filter.d/xinetd-fail.conf b/config/filter.d/xinetd-fail.conf index d75e3d66..b4093d98 100644 --- a/config/filter.d/xinetd-fail.conf +++ b/config/filter.d/xinetd-fail.conf @@ -14,8 +14,10 @@ before = common.conf _daemon = xinetd -failregex = ^%(__prefix_line)sFAIL: \S+ address from=<HOST>$ - ^%(__prefix_line)sFAIL: \S+ libwrap from=<HOST>$ +prefregex = ^%(__prefix_line)sFAIL: <F-CONTENT>.+</F-CONTENT>$ + +failregex = ^\S+ address from=<HOST>$ + ^\S+ libwrap from=<HOST>$ ignoreregex = diff --git a/config/jail.conf b/config/jail.conf index b7c927e2..e3e89ff0 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -44,10 +44,14 @@ before = paths-debian.conf # MISCELLANEOUS OPTIONS # -# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not -# ban a host which matches an address in this list. Several addresses can be -# defined using space (and/or comma) separator. -ignoreip = 127.0.0.1/8 ::1 +# "ignorself" specifies whether the local resp. own IP addresses should be ignored +# (default is true). Fail2ban will not ban a host which matches such addresses. +#ignorself = true + +# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban +# will not ban a host which matches an address in this list. Several addresses +# can be defined using space (and/or comma) separator. +#ignoreip = 127.0.0.1/8 ::1 # External command that will take an tagged arguments to ignore, e.g. <ip>, # and return true if the IP is to be ignored. False otherwise. @@ -130,7 +134,7 @@ filter = %(__name__)s destemail = root@localhost # Sender email address used solely for some actions -sender = root@localhost +sender = root@<fq-hostname> # E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the # mailing. Change mta configuration parameter to mail if you want to @@ -207,6 +211,12 @@ action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", ag # action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"] +# Report ban via abuseipdb.com. +# +# See action.d/abuseipdb.conf for usage example and details. +# +action_abuseipdb = abuseipdb + # Choose default action. To change, just override value of 'action' with the # interpolation to the chosen action shortcut (e.g. action_mw, action_mwl, etc) in jail.local # globally (section [DEFAULT]) or per specific section @@ -223,15 +233,11 @@ action = %(action_)s [sshd] -port = ssh -logpath = %(sshd_log)s -backend = %(sshd_backend)s - - -[sshd-ddos] -# This jail corresponds to the standard configuration in Fail2ban. -# The mail-whois action send a notification e-mail with a whois request -# in the body. +# To use more aggressive sshd modes set filter parameter "mode" in jail.local: +# normal (default), ddos, extra or aggressive (combines all). +# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details. +mode = normal +filter = sshd[mode=%(mode)s] port = ssh logpath = %(sshd_log)s backend = %(sshd_backend)s @@ -377,6 +383,8 @@ logpath = %(lighttpd_error_log)s port = http,https logpath = %(roundcube_errors_log)s +# Use following line in your jail.local if roundcube logs to journal. +#backend = %(syslog_backend)s [openwebmail] @@ -525,14 +533,17 @@ backend = %(syslog_backend)s [postfix] - -port = smtp,465,submission -logpath = %(postfix_log)s -backend = %(postfix_backend)s +# To use another modes set filter parameter "mode" in jail.local: +mode = more +filter = postfix[mode=%(mode)s] +port = smtp,465,submission +logpath = %(postfix_log)s +backend = %(postfix_backend)s [postfix-rbl] +filter = postfix[mode=rbl] port = smtp,465,submission logpath = %(postfix_log)s backend = %(postfix_backend)s @@ -547,7 +558,11 @@ backend = %(syslog_backend)s [sendmail-reject] - +# To use more aggressive modes set filter parameter "mode" in jail.local: +# normal (default), extra or aggressive +# See "tests/files/logs/sendmail-reject" or "filter.d/sendmail-reject.conf" for usage example and details. +mode = normal +filter = sendmail-reject[mode=%(mode)s] port = smtp,465,submission logpath = %(syslog_mail)s backend = %(syslog_backend)s @@ -614,6 +629,7 @@ backend = %(syslog_backend)s [postfix-sasl] +filter = postfix[mode=auth] port = smtp,465,submission,imap3,imaps,pop3,pop3s # You might consider monitoring /var/log/mail.warn instead if you are # running postfix since it would provide the same log lines at the @@ -731,6 +747,13 @@ logpath = %(mysql_log)s backend = %(mysql_backend)s +# Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf') +[mongodb-auth] +# change port when running with "--shardsvr" or "--configsvr" runtime operation +port = 27017 +logpath = /var/log/mongodb/mongodb.log + + # Jail for more extended banning of persistent abusers # !!! WARNINGS !!! # 1. Make sure that your loglevel specified in fail2ban.conf/.local @@ -810,8 +833,9 @@ maxretry = 1 [pass2allow-ftp] # this pass2allow example allows FTP traffic after successful HTTP authentication port = ftp,ftp-data,ftps,ftps-data -# knocking_url variable must be overridden to some secret value in filter.d/apache-pass.local -filter = apache-pass +# knocking_url variable must be overridden to some secret value in jail.local +knocking_url = /knocking/ +filter = apache-pass[knocking_url="%(knocking_url)s"] # access log of the website with HTTP auth logpath = %(apache_access_log)s blocktype = RETURN @@ -845,3 +869,8 @@ logpath = /var/log/haproxy.log port = ldap,ldaps filter = slapd logpath = /var/log/slapd.log + +[domino-smtp] +port = smtp,ssmtp +filter = domino-smtp +logpath = /home/domino01/data/IBM_TECHNICAL_SUPPORT/console.log diff --git a/config/paths-arch.conf b/config/paths-arch.conf new file mode 100644 index 00000000..270ce65d --- /dev/null +++ b/config/paths-arch.conf @@ -0,0 +1,32 @@ +# Arch + +[INCLUDES] + +before = paths-common.conf + +after = paths-overrides.local + + +[DEFAULT] + +apache_error_log = /var/log/httpd/*error_log + +apache_access_log = /var/log/httpd/*access_log + +exim_main_log = /var/log/exim/main.log + +mysql_log = /var/log/mariadb/mariadb.log + /var/log/mysqld.log + +roundcube_errors_log = /var/log/roundcubemail/errors + +# These services will log to the journal via syslog, so use the journal by +# default. +syslog_backend = systemd +sshd_backend = systemd +dropbear_backend = systemd +proftpd_backend = systemd +pureftpd_backend = systemd +wuftpd_backend = systemd +postfix_backend = systemd +dovecot_backend = systemd diff --git a/config/paths-common.conf b/config/paths-common.conf index 9072136c..51323d6b 100644 --- a/config/paths-common.conf +++ b/config/paths-common.conf @@ -7,7 +7,7 @@ after = paths-overrides.local [DEFAULT] -default_backend = auto +default_backend = %(default/backend)s sshd_log = %(syslog_authpriv)s sshd_backend = %(default_backend)s diff --git a/config/paths-debian.conf b/config/paths-debian.conf index 4c27dac8..1ca4d242 100644 --- a/config/paths-debian.conf +++ b/config/paths-debian.conf @@ -11,7 +11,9 @@ after = paths-overrides.local syslog_mail = /var/log/mail.log -syslog_mail_warn = /var/log/mail.warn +# control the `mail.warn` setting, see `/etc/rsyslog.d/50-default.conf` (if commented `mail.*` wins). +# syslog_mail_warn = /var/log/mail.warn +syslog_mail_warn = %(syslog_mail)s syslog_authpriv = /var/log/auth.log diff --git a/config/paths-freebsd.conf b/config/paths-freebsd.conf index fd788ab4..91b23636 100644 --- a/config/paths-freebsd.conf +++ b/config/paths-freebsd.conf @@ -34,13 +34,13 @@ auditd_log = /dev/null # http://svnweb.freebsd.org/ports/head/www/apache24/files/patch-config.layout # http://svnweb.freebsd.org/ports/head/www/apache22/files/patch-config.layout -apache_error_log = /usr/local/www/logs/*error[_.]log +apache_error_log = /var/log/httpd-error.log -apache_access_log = /usr/local/www/logs/*access[_.]log +apache_access_log = /var/log/httpd-access.log # http://svnweb.freebsd.org/ports/head/www/nginx/Makefile?view=markup -nginx_error_log = /var/log/nginx-error.log +nginx_error_log = /var/log/nginx/error.log -nginx_access_log = /var/log/nginx-access.log +nginx_access_log = /var/log/nginx/access.log diff --git a/config/paths-opensuse.conf b/config/paths-opensuse.conf index 0d6ad522..227a5e98 100644 --- a/config/paths-opensuse.conf +++ b/config/paths-opensuse.conf @@ -36,3 +36,15 @@ mysql_log = /var/log/mysql/mysqld.log roundcube_errors_log = /srv/www/roundcubemail/logs/errors solidpop3d_log = %(syslog_mail)s + +# These services will log to the journal via syslog, so use the journal by +# default. +syslog_backend = systemd +sshd_backend = systemd +dropbear_backend = systemd +proftpd_backend = systemd +pureftpd_backend = systemd +wuftpd_backend = systemd +postfix_backend = systemd +dovecot_backend = systemd +mysql_backend = systemd diff --git a/fail2ban/__init__.py b/fail2ban/__init__.py index 0f0fc3ec..317f53e7 100644 --- a/fail2ban/__init__.py +++ b/fail2ban/__init__.py @@ -27,8 +27,10 @@ __license__ = "GPL" import logging.handlers # Custom debug levels +logging.MSG = logging.INFO - 2 logging.TRACEDEBUG = 7 logging.HEAVYDEBUG = 5 +logging.addLevelName(logging.MSG, 'MSG') logging.addLevelName(logging.TRACEDEBUG, 'TRACE') logging.addLevelName(logging.HEAVYDEBUG, 'HEAVY') diff --git a/fail2ban/client/actionreader.py b/fail2ban/client/actionreader.py index e5025fa3..ace0b898 100644 --- a/fail2ban/client/actionreader.py +++ b/fail2ban/client/actionreader.py @@ -28,6 +28,7 @@ import os from .configreader import DefinitionInitConfigReader from ..helpers import getLogger +from ..server.action import CommandAction # Gets the instance of the logger. logSys = getLogger(__name__) @@ -37,16 +38,23 @@ class ActionReader(DefinitionInitConfigReader): _configOpts = { "actionstart": ["string", None], + "actionstart_on_demand": ["string", None], "actionstop": ["string", None], + "actionflush": ["string", None], "actionreload": ["string", None], "actioncheck": ["string", None], "actionrepair": ["string", None], "actionban": ["string", None], "actionunban": ["string", None], + "norestored": ["string", None], } def __init__(self, file_, jailName, initOpts, **kwargs): - self._name = initOpts.get("actname", file_) + actname = initOpts.get("actname") + if actname is None: + actname = file_ + initOpts["actname"] = actname + self._name = actname DefinitionInitConfigReader.__init__( self, file_, jailName, initOpts, **kwargs) @@ -64,16 +72,25 @@ class ActionReader(DefinitionInitConfigReader): return self._name def convert(self): + opts = self.getCombined( + ignore=CommandAction._escapedTags | set(('timeout', 'bantime'))) + # type-convert only after combined (otherwise boolean converting prevents substitution): + for o in ('norestored', 'actionstart_on_demand'): + if opts.get(o): + opts[o] = self._convert_to_boolean(opts[o]) + + # stream-convert: head = ["set", self._jailName] stream = list() stream.append(head + ["addaction", self._name]) multi = [] - for opt, optval in self._opts.iteritems(): + for opt, optval in opts.iteritems(): if opt in self._configOpts: multi.append([opt, optval]) if self._initOpts: for opt, optval in self._initOpts.iteritems(): - multi.append([opt, optval]) + if opt not in self._configOpts: + multi.append([opt, optval]) if len(multi) > 1: stream.append(["multi-set", self._jailName, "action", self._name, multi]) elif len(multi): diff --git a/fail2ban/client/beautifier.py b/fail2ban/client/beautifier.py index df44afbb..4d9e549f 100644 --- a/fail2ban/client/beautifier.py +++ b/fail2ban/client/beautifier.py @@ -89,6 +89,8 @@ class Beautifier: val = " ".join(map(str, res1[1])) if isinstance(res1[1], list) else res1[1] msg.append("%s %s:\t%s" % (prefix1, res1[0], val)) msg = "\n".join(msg) + elif len(inC) < 2: + pass # to few cmd args for below elif inC[1] == "syslogsocket": msg = "Current syslog socket is:\n" msg += "`- " + response @@ -110,6 +112,8 @@ class Beautifier: else: msg = "Current database purge age is:\n" msg += "`- %iseconds" % response + elif len(inC) < 3: + pass # to few cmd args for below elif inC[2] in ("logpath", "addlogpath", "dellogpath"): if len(response) == 0: msg = "No file is currently monitored" @@ -178,7 +182,8 @@ class Beautifier: msg += ", ".join(response) except Exception: logSys.warning("Beautifier error. Please report the error") - logSys.error("Beautify %r with %r failed", response, self.__inputCmd) + logSys.error("Beautify %r with %r failed", response, self.__inputCmd, + exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) msg = repr(msg) + repr(response) return msg diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py index 35fa7498..b626be9b 100644 --- a/fail2ban/client/configparserinc.py +++ b/fail2ban/client/configparserinc.py @@ -32,8 +32,8 @@ from ..helpers import getLogger if sys.version_info >= (3,2): # SafeConfigParser deprecated from Python 3.2 (renamed to ConfigParser) - from configparser import ConfigParser as SafeConfigParser, \ - BasicInterpolation + from configparser import ConfigParser as SafeConfigParser, BasicInterpolation, \ + InterpolationMissingOptionError, NoSectionError # And interpolation of __name__ was simply removed, thus we need to # decorate default interpolator to handle it @@ -52,20 +52,32 @@ if sys.version_info >= (3,2): But should be fine to reincarnate for our use case """ def _interpolate_some(self, parser, option, accum, rest, section, map, - depth): + *args, **kwargs): if section and not (__name__ in map): map = map.copy() # just to be safe map['__name__'] = section - return super(BasicInterpolationWithName, self)._interpolate_some( - parser, option, accum, rest, section, map, depth) + # try to wrap section options like %(section/option)s: + parser._map_section_options(section, option, rest, map) + return super(BasicInterpolationWithName, self)._interpolate_some( + parser, option, accum, rest, section, map, *args, **kwargs) else: # pragma: no cover - from ConfigParser import SafeConfigParser + from ConfigParser import SafeConfigParser, \ + InterpolationMissingOptionError, NoSectionError + + # Interpolate missing known/option as option from default section + SafeConfigParser._cp_interpolate_some = SafeConfigParser._interpolate_some + def _interpolate_some(self, option, accum, rest, section, map, *args, **kwargs): + # try to wrap section options like %(section/option)s: + self._map_section_options(section, option, rest, map) + return self._cp_interpolate_some(option, accum, rest, section, map, *args, **kwargs) + SafeConfigParser._interpolate_some = _interpolate_some # Gets the instance of the logger. logSys = getLogger(__name__) logLevel = 7 + __all__ = ['SafeConfigParserWithIncludes'] @@ -100,6 +112,8 @@ after = 1.conf SECTION_NAME = "INCLUDES" + SECTION_OPTSUBST_CRE = re.compile(r'%\(([\w\-]+/([^\)]+))\)s') + CONDITIONAL_RE = re.compile(r"^(\w+)(\?.+)$") if sys.version_info >= (3,2): @@ -117,6 +131,46 @@ after = 1.conf SafeConfigParser.__init__(self, *args, **kwargs) self._cfg_share = share_config + def _map_section_options(self, section, option, rest, map): + """ + Interpolates values of the section options (name syntax `%(section/option)s`). + + Fallback: try to wrap missing default options as "default/options" resp. "known/options" + """ + if '/' not in rest or '%(' not in rest: # pragma: no cover + return 0 + soptrep = SafeConfigParserWithIncludes.SECTION_OPTSUBST_CRE.findall(rest) + if not soptrep: # pragma: no cover + return 0 + for sopt, opt in soptrep: + if sopt not in map: + sec = sopt[:~len(opt)] + seclwr = sec.lower() + if seclwr != 'default': + if seclwr == 'known': + # try get raw value from known options: + try: + v = self._sections['KNOWN'][opt] + except KeyError: + # fallback to default: + try: + v = self._defaults[opt] + except KeyError: # pragma: no cover + continue + else: + # get raw value of opt in section: + v = self.get(sec, opt, raw=True) + else: + try: + v = self._defaults[opt] + except KeyError: # pragma: no cover + continue + self._defaults[sopt] = v + try: # for some python versions need to duplicate it in map-vars also: + map[sopt] = v + except: pass + return 1 + @property def share_config(self): return self._cfg_share @@ -200,6 +254,21 @@ after = 1.conf def get_sections(self): return self._sections + def options(self, section, withDefault=True): + """Return a list of option names for the given section name. + + Parameter `withDefault` controls the include of names from section `[DEFAULT]` + """ + try: + opts = self._sections[section] + except KeyError: # pragma: no cover + raise NoSectionError(section) + if withDefault: + # mix it with defaults: + return set(opts.keys()) | set(self._defaults) + # only own option names: + return opts.keys() + def read(self, filenames, get_includes=True): if not isinstance(filenames, list): filenames = [ filenames ] @@ -244,11 +313,7 @@ after = 1.conf s2 = alls.get(n) if isinstance(s2, dict): # save previous known values, for possible using in local interpolations later: - sk = {} - for k, v in s2.iteritems(): - if not k.startswith('known/') and k != '__name__': - sk['known/'+k] = v - s2.update(sk) + self.merge_section('KNOWN', s2, '') # merge section s2.update(s) else: @@ -265,14 +330,18 @@ after = 1.conf else: return SafeConfigParser.read(self, fileNamesFull) - def merge_section(self, section, options, pref='known/'): + def merge_section(self, section, options, pref=None): alls = self.get_sections() - if pref == '': - alls[section].update(options) + try: + sec = alls[section] + except KeyError: + alls[section] = sec = dict() + if not pref: + sec.update(options) return sk = {} for k, v in options.iteritems(): if not k.startswith(pref) and k != '__name__': sk[pref+k] = v - alls[section].update(sk) + sec.update(sk) diff --git a/fail2ban/client/configreader.py b/fail2ban/client/configreader.py index a72ca1e9..381af759 100644 --- a/fail2ban/client/configreader.py +++ b/fail2ban/client/configreader.py @@ -29,24 +29,12 @@ import os from ConfigParser import NoOptionError, NoSectionError from .configparserinc import sys, SafeConfigParserWithIncludes, logLevel -from ..helpers import getLogger +from ..helpers import getLogger, _merge_dicts, substituteRecursiveTags # Gets the instance of the logger. logSys = getLogger(__name__) -# if sys.version_info >= (3,5): -# def _merge_dicts(x, y): -# return {**x, **y} -# else: -def _merge_dicts(x, y): - r = x - if y: - r = x.copy() - r.update(y) - return r - - class ConfigReader(): """Generic config reader class. @@ -121,33 +109,44 @@ class ConfigReader(): self._cfg = ConfigReaderUnshared(**self._cfg_share_kwargs) def sections(self): - if self._cfg is not None: - return self._cfg.sections() - return [] + try: + return (n for n in self._cfg.sections() if n != 'KNOWN') + except AttributeError: + return [] def has_section(self, sec): - if self._cfg is not None: + try: return self._cfg.has_section(sec) - return False + except AttributeError: + return False - def merge_section(self, *args, **kwargs): - if self._cfg is not None: - return self._cfg.merge_section(*args, **kwargs) + def merge_section(self, section, *args, **kwargs): + try: + return self._cfg.merge_section(section, *args, **kwargs) + except AttributeError: + raise NoSectionError(section) + + def options(self, section, withDefault=False): + """Return a list of option names for the given section name. - def options(self, *args): - if self._cfg is not None: - return self._cfg.options(*args) - return {} + Parameter `withDefault` controls the include of names from section `[DEFAULT]` + """ + try: + return self._cfg.options(section, withDefault) + except AttributeError: + raise NoSectionError(section) def get(self, sec, opt, raw=False, vars={}): - if self._cfg is not None: + try: return self._cfg.get(sec, opt, raw=raw, vars=vars) - return None + except AttributeError: + raise NoSectionError(sec) - def getOptions(self, *args, **kwargs): - if self._cfg is not None: - return self._cfg.getOptions(*args, **kwargs) - return {} + def getOptions(self, section, *args, **kwargs): + try: + return self._cfg.getOptions(section, *args, **kwargs) + except AttributeError: + raise NoSectionError(section) class ConfigReaderUnshared(SafeConfigParserWithIncludes): @@ -176,6 +175,8 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes): if not os.path.exists(self._basedir): raise ValueError("Base configuration directory %s does not exist " % self._basedir) + if filename.startswith("./"): # pragma: no cover + filename = os.path.abspath(filename) basename = os.path.join(self._basedir, filename) logSys.debug("Reading configs for %s under %s " , filename, self._basedir) config_files = [ basename + ".conf" ] @@ -224,6 +225,7 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes): values = dict() if pOptions is None: pOptions = {} + # Get only specified options: for optname in options: if isinstance(options, (list,tuple)): if len(optname) > 2: @@ -276,9 +278,13 @@ class DefinitionInitConfigReader(ConfigReader): def __init__(self, file_, jailName, initOpts, **kwargs): ConfigReader.__init__(self, **kwargs) + if file_.startswith("./"): # pragma: no cover + file_ = os.path.abspath(file_) self.setFile(file_) self.setJailName(jailName) self._initOpts = initOpts + self._pOpts = dict() + self._defCache = dict() def setFile(self, fileName): self._file = fileName @@ -302,23 +308,74 @@ class DefinitionInitConfigReader(ConfigReader): self._create_unshared(self._file) return SafeConfigParserWithIncludes.read(self._cfg, self._file) - def getOptions(self, pOpts): + def getOptions(self, pOpts, all=False): # overwrite static definition options with init values, supplied as # direct parameters from jail-config via action[xtra1="...", xtra2=...]: + if not pOpts: + pOpts = dict() if self._initOpts: - if not pOpts: - pOpts = dict() pOpts = _merge_dicts(pOpts, self._initOpts) self._opts = ConfigReader.getOptions( self, "Definition", self._configOpts, pOpts) - + self._pOpts = pOpts if self.has_section("Init"): - for opt in self.options("Init"): - v = self.get("Init", opt) - if not opt.startswith('known/') and opt != '__name__': + # get only own options (without options from default): + getopt = lambda opt: self.get("Init", opt) + for opt in self.options("Init", withDefault=False): + if opt == '__name__': continue + v = None + if not opt.startswith('known/'): + if v is None: v = getopt(opt) self._initOpts['known/'+opt] = v - if not opt in self._initOpts: + if opt not in self._initOpts: + if v is None: v = getopt(opt) self._initOpts[opt] = v + if all and self.has_section("Definition"): + # merge with all definition options (and options from default), + # bypass already converted option (so merge only new options): + for opt in self.options("Definition"): + if opt == '__name__' or opt in self._opts: continue + self._opts[opt] = self.get("Definition", opt) + + + def _convert_to_boolean(self, value): + return value.lower() in ("1", "yes", "true", "on") + + def getCombOption(self, optname): + """Get combined definition option (as string) using pre-set and init + options as preselection (values with higher precedence as specified in section). + + Can be used only after calling of getOptions. + """ + try: + return self._defCache[optname] + except KeyError: + try: + v = self.get("Definition", optname, vars=self._pOpts) + except (NoSectionError, NoOptionError, ValueError): + v = None + self._defCache[optname] = v + return v + + def getCombined(self, ignore=()): + combinedopts = self._opts + if self._initOpts: + combinedopts = _merge_dicts(combinedopts, self._initOpts) + if not len(combinedopts): + return {} + # ignore conditional options: + ignore = set(ignore).copy() + for n in combinedopts: + cond = SafeConfigParserWithIncludes.CONDITIONAL_RE.match(n) + if cond: + n, cond = cond.groups() + ignore.add(n) + # substiture options already specified direct: + opts = substituteRecursiveTags(combinedopts, + ignore=ignore, addrepl=self.getCombOption) + if not opts: + raise ValueError('recursive tag definitions unable to be resolved') + return opts def convert(self): raise NotImplementedError diff --git a/fail2ban/client/fail2bancmdline.py b/fail2ban/client/fail2bancmdline.py index a9387bc6..f213d037 100644 --- a/fail2ban/client/fail2bancmdline.py +++ b/fail2ban/client/fail2bancmdline.py @@ -111,6 +111,7 @@ class Fail2banCmdLine(): output(" -f start server in foreground") output(" --async start server in async mode (for internal usage only, don't read configuration)") output(" --timeout timeout to wait for the server (for internal usage only, don't read configuration)") + output(" --str2sec <STRING> convert time abbreviation format to seconds") output(" -h, --help display this help message") output(" -V, --version print the version") @@ -155,9 +156,13 @@ class Fail2banCmdLine(): self._conf["background"] = False elif o == "--async": self._conf["async"] = True - elif o == "-timeout": - from ..mytime import MyTime + elif o == "--timeout": + from ..server.mytime import MyTime self._conf["timeout"] = MyTime.str2seconds(opt[1]) + elif o == "--str2sec": + from ..server.mytime import MyTime + output(MyTime.str2seconds(opt[1])) + return True elif o in ["-h", "--help"]: self.dispUsage() return True @@ -179,7 +184,7 @@ class Fail2banCmdLine(): # Reads the command line options. try: cmdOpts = 'hc:s:p:xfbdtviqV' - cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async', 'timeout=', 'help', 'version'] + cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async', 'timeout=', 'str2sec=', 'help', 'version'] optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts) except getopt.GetoptError: self.dispUsage() diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index ff55c417..89225ffb 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -41,27 +41,30 @@ from optparse import OptionParser, Option from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError try: # pragma: no cover - from systemd import journal from ..server.filtersystemd import FilterSystemd except ImportError: - journal = None + FilterSystemd = None from ..version import version +from .jailreader import JailReader from .filterreader import FilterReader from ..server.filter import Filter, FileContainer -from ..server.failregex import RegexException +from ..server.failregex import Regex, RegexException from ..helpers import str2LogLevel, getVerbosityFormat, FormatterWithTraceBack, getLogger, PREFER_ENC # Gets the instance of the logger. logSys = getLogger("fail2ban") -def debuggexURL(sample, regex): - q = urllib.urlencode({ 're': regex.replace('<HOST>', '(?&.ipv4)'), - 'str': sample, - 'flavor': 'python' }) - return 'https://www.debuggex.com/?' + q +def debuggexURL(sample, regex, multiline=False, useDns="yes"): + args = { + 're': Regex._resolveHostTag(regex, useDns=useDns), + 'str': sample, + 'flavor': 'python' + } + if multiline: args['flags'] = 'm' + return 'https://www.debuggex.com/?' + urllib.urlencode(args) -def output(args): +def output(args): # pragma: no cover (overriden in test-cases) print(args) def shortstr(s, l=53): @@ -80,7 +83,7 @@ def pprint_list(l, header=None): s = '' output( s + "| " + "\n| ".join(l) + '\n`-' ) -def journal_lines_gen(myjournal): # pragma: no cover +def journal_lines_gen(flt, myjournal): # pragma: no cover while True: try: entry = myjournal.get_next() @@ -88,7 +91,7 @@ def journal_lines_gen(myjournal): # pragma: no cover continue if not entry: break - yield FilterSystemd.formatJournalEntry(entry) + yield flt.formatJournalEntry(entry) def get_opt_parser(): # use module docstring for help output @@ -120,8 +123,12 @@ Report bugs to https://github.com/fail2ban/fail2ban/issues version="%prog " + version) p.add_options([ + Option("-c", "--config", default='/etc/fail2ban', + help="set alternate config directory"), Option("-d", "--datepattern", help="set custom pattern used to match date/times"), + Option("--timezone", "--TZ", action='store', default=None, + help="set time-zone used by convert time format"), Option("-e", "--encoding", default=PREFER_ENC, help="File encoding. Default: system locale"), Option("-r", "--raw", action='store_true', default=False, @@ -196,15 +203,17 @@ class RegexStat(object): class LineStats(object): """Just a convenience container for stats """ - def __init__(self): + def __init__(self, opts): self.tested = self.matched = 0 self.matched_lines = [] self.missed = 0 self.missed_lines = [] - self.missed_lines_timeextracted = [] self.ignored = 0 self.ignored_lines = [] - self.ignored_lines_timeextracted = [] + if opts.debuggex: + self.matched_lines_timeextracted = [] + self.missed_lines_timeextracted = [] + self.ignored_lines_timeextracted = [] def __str__(self): return "%(tested)d lines, %(ignored)d ignored, %(matched)d matched, %(missed)d missed" % self @@ -228,14 +237,16 @@ class Fail2banRegex(object): self._ignoreregex = list() self._failregex = list() self._time_elapsed = None - self._line_stats = LineStats() + self._line_stats = LineStats(opts) if opts.maxlines: self.setMaxLines(opts.maxlines) else: self._maxlines = 20 if opts.journalmatch is not None: - self.setJournalMatch(opts.journalmatch.split()) + self.setJournalMatch(shlex.split(opts.journalmatch)) + if opts.timezone: + self._filter.setLogTimeZone(opts.timezone) if opts.datepattern: self.setDatePattern(opts.datepattern) if opts.usedns: @@ -243,6 +254,7 @@ class Fail2banRegex(object): self._filter.returnRawHost = opts.raw self._filter.checkFindTime = False self._filter.checkAllRegex = True + self._opts = opts def decode_line(self, line): return FileContainer.decode_line('<LOG>', self._encoding, line) @@ -265,69 +277,117 @@ class Fail2banRegex(object): output( "Use maxlines : %d" % self._filter.getMaxLines() ) def setJournalMatch(self, v): - if self._journalmatch is None: - self._journalmatch = v + self._journalmatch = v def readRegex(self, value, regextype): assert(regextype in ('fail', 'ignore')) regex = regextype + 'regex' - if os.path.isfile(value) or os.path.isfile(value + '.conf'): - if os.path.basename(os.path.dirname(value)) == 'filter.d': + # try to check - we've case filter?[options...]?: + basedir = self._opts.config + fltFile = None + fltOpt = {} + if regextype == 'fail': + fltName, fltOpt = JailReader.extractOptions(value) + if fltName is not None: + if "." in fltName[~5:]: + tryNames = (fltName,) + else: + tryNames = (fltName, fltName + '.conf', fltName + '.local') + for fltFile in tryNames: + if not "/" in fltFile: + if os.path.basename(basedir) == 'filter.d': + fltFile = os.path.join(basedir, fltFile) + else: + fltFile = os.path.join(basedir, 'filter.d', fltFile) + else: + basedir = os.path.dirname(fltFile) + if os.path.isfile(fltFile): + break + fltFile = None + # if it is filter file: + if fltFile is not None: + if (basedir == self._opts.config + or os.path.basename(basedir) == 'filter.d' + or ("." not in fltName[~5:] and "/" not in fltName) + ): ## within filter.d folder - use standard loading algorithm to load filter completely (with .local etc.): - basedir = os.path.dirname(os.path.dirname(value)) - value = os.path.splitext(os.path.basename(value))[0] - output( "Use %11s filter file : %s, basedir: %s" % (regex, value, basedir) ) - reader = FilterReader(value, 'fail2ban-regex-jail', {}, share_config=self.share_config, basedir=basedir) - if not reader.read(): - output( "ERROR: failed to load filter %s" % value ) - return False + if os.path.basename(basedir) == 'filter.d': + basedir = os.path.dirname(basedir) + fltName = os.path.splitext(os.path.basename(fltName))[0] + output( "Use %11s filter file : %s, basedir: %s" % (regex, fltName, basedir) ) else: ## foreign file - readexplicit this file and includes if possible: - output( "Use %11s file : %s" % (regex, value) ) - reader = FilterReader(value, 'fail2ban-regex-jail', {}, share_config=self.share_config) - reader.setBaseDir(None) - if not reader.readexplicit(): - output( "ERROR: failed to read %s" % value ) - return False + output( "Use %11s file : %s" % (regex, fltName) ) + basedir = None + if fltOpt: + output( "Use filter options : %r" % fltOpt ) + reader = FilterReader(fltName, 'fail2ban-regex-jail', fltOpt, share_config=self.share_config, basedir=basedir) + ret = None + try: + if basedir is not None: + ret = reader.read() + else: + ## foreign file - readexplicit this file and includes if possible: + reader.setBaseDir(None) + ret = reader.readexplicit() + except Exception as e: + output("Wrong config file: %s" % (str(e),)) + if self._verbose: raise(e) + if not ret: + output( "ERROR: failed to load filter %s" % value ) + return False reader.getOptions(None) readercommands = reader.convert() - regex_values = [ - RegexStat(m[3]) - for m in filter( - lambda x: x[0] == 'set' and x[2] == "add%sregex" % regextype, - readercommands) - ] + [ - RegexStat(m) - for mm in filter( - lambda x: x[0] == 'multi-set' and x[2] == "add%sregex" % regextype, - readercommands) - for m in mm[3] - ] - # Read out and set possible value of maxlines - for command in readercommands: - if command[2] == "maxlines": - maxlines = int(command[3]) - try: - self.setMaxLines(maxlines) - except ValueError: - output( "ERROR: Invalid value for maxlines (%(maxlines)r) " \ - "read from %(value)s" % locals() ) - return False - elif command[2] == 'addjournalmatch': - journalmatch = command[3:] - self.setJournalMatch(journalmatch) - elif command[2] == 'datepattern': - datepattern = command[3] - self.setDatePattern(datepattern) + + regex_values = {} + for opt in readercommands: + if opt[0] == 'multi-set': + optval = opt[3] + elif opt[0] == 'set': + optval = opt[3:] + else: # pragma: no cover + continue + try: + if opt[2] == "prefregex": + for optval in optval: + self._filter.prefRegex = optval + elif opt[2] == "addfailregex": + stor = regex_values.get('fail') + if not stor: stor = regex_values['fail'] = list() + for optval in optval: + stor.append(RegexStat(optval)) + #self._filter.addFailRegex(optval) + elif opt[2] == "addignoreregex": + stor = regex_values.get('ignore') + if not stor: stor = regex_values['ignore'] = list() + for optval in optval: + stor.append(RegexStat(optval)) + #self._filter.addIgnoreRegex(optval) + elif opt[2] == "maxlines": + for optval in optval: + self.setMaxLines(optval) + elif opt[2] == "datepattern": + for optval in optval: + self.setDatePattern(optval) + elif opt[2] == "addjournalmatch": # pragma: no cover + if self._opts.journalmatch is None: + self.setJournalMatch(optval) + except ValueError as e: # pragma: no cover + output( "ERROR: Invalid value for %s (%r) " \ + "read from %s: %s" % (opt[2], optval, value, e) ) + return False + else: output( "Use %11s line : %s" % (regex, shortstr(value)) ) - regex_values = [RegexStat(value)] - - setattr(self, "_" + regex, regex_values) - for regex in regex_values: - getattr( - self._filter, - 'add%sRegex' % regextype.title())(regex.getFailRegex()) + regex_values = {regextype: [RegexStat(value)]} + + for regextype, regex_values in regex_values.iteritems(): + regex = regextype + 'regex' + setattr(self, "_" + regex, regex_values) + for regex in regex_values: + getattr( + self._filter, + 'add%sRegex' % regextype.title())(regex.getFailRegex()) return True def testIgnoreRegex(self, line): @@ -337,7 +397,7 @@ class Fail2banRegex(object): if ret is not None: found = True regex = self._ignoreregex[ret].inc() - except RegexException as e: + except RegexException as e: # pragma: no cover output( 'ERROR: %s' % e ) return False return found @@ -347,6 +407,7 @@ class Fail2banRegex(object): fullBuffer = len(orgLineBuffer) >= self._filter.getMaxLines() try: ret = self._filter.processLine(line, date) + lines = [] line = self._filter.processedLine() for match in ret: # Append True/False flag depending if line was matched by @@ -355,7 +416,7 @@ class Fail2banRegex(object): regex = self._failregex[match[0]] regex.inc() regex.appendIP(match) - except RegexException as e: + except RegexException as e: # pragma: no cover output( 'ERROR: %s' % e ) return False for bufLine in orgLineBuffer[int(fullBuffer):]: @@ -363,14 +424,23 @@ class Fail2banRegex(object): try: self._line_stats.missed_lines.pop( self._line_stats.missed_lines.index("".join(bufLine))) - self._line_stats.missed_lines_timeextracted.pop( - self._line_stats.missed_lines_timeextracted.index( - "".join(bufLine[::2]))) + if self._debuggex: + self._line_stats.missed_lines_timeextracted.pop( + self._line_stats.missed_lines_timeextracted.index( + "".join(bufLine[::2]))) except ValueError: pass - else: - self._line_stats.matched += 1 - self._line_stats.missed -= 1 + # if buffering - add also another lines from match: + if self._print_all_matched: + if not self._debuggex: + self._line_stats.matched_lines.append("".join(bufLine)) + else: + lines.append(bufLine[0] + bufLine[2]) + self._line_stats.matched += 1 + self._line_stats.missed -= 1 + if lines: # pre-lines parsed in multiline mode (buffering) + lines.append(line) + line = "\n".join(lines) return line, ret def process(self, test_lines): @@ -392,19 +462,23 @@ class Fail2banRegex(object): self._line_stats.ignored += 1 if not self._print_no_ignored and (self._print_all_ignored or self._line_stats.ignored <= self._maxlines + 1): self._line_stats.ignored_lines.append(line) - self._line_stats.ignored_lines_timeextracted.append(line_datetimestripped) + if self._debuggex: + self._line_stats.ignored_lines_timeextracted.append(line_datetimestripped) if len(ret) > 0: assert(not is_ignored) self._line_stats.matched += 1 if self._print_all_matched: self._line_stats.matched_lines.append(line) + if self._debuggex: + self._line_stats.matched_lines_timeextracted.append(line_datetimestripped) else: if not is_ignored: self._line_stats.missed += 1 if not self._print_no_missed and (self._print_all_missed or self._line_stats.missed <= self._maxlines + 1): self._line_stats.missed_lines.append(line) - self._line_stats.missed_lines_timeextracted.append(line_datetimestripped) + if self._debuggex: + self._line_stats.missed_lines_timeextracted.append(line_datetimestripped) self._line_stats.tested += 1 self._time_elapsed = time.time() - t0 @@ -414,6 +488,7 @@ class Fail2banRegex(object): assert(self._line_stats.missed == lstats.tested - (lstats.matched + lstats.ignored)) lines = lstats[ltype] l = lstats[ltype + '_lines'] + multiline = self._filter.getMaxLines() > 1 if lines: header = "%s line(s):" % (ltype.capitalize(),) if self._debuggex: @@ -427,7 +502,8 @@ class Fail2banRegex(object): for arg in [l, regexlist]: ans = [ x + [y] for x in ans for y in arg ] b = map(lambda a: a[0] + ' | ' + a[1].getFailRegex() + ' | ' + - debuggexURL(self.encode_line(a[0]), a[1].getFailRegex()), ans) + debuggexURL(self.encode_line(a[0]), a[1].getFailRegex(), + multiline, self._opts.usedns), ans) pprint_list([x.rstrip() for x in b], header) else: output( "%s too many to print. Use --print-all-%s " \ @@ -502,14 +578,14 @@ class Fail2banRegex(object): for line in hdlr: yield self.decode_line(line) - def start(self, opts, args): + def start(self, args): cmd_log, cmd_regex = args[:2] try: - if not self.readRegex(cmd_regex, 'fail'): + if not self.readRegex(cmd_regex, 'fail'): # pragma: no cover return False - if len(args) == 3 and not self.readRegex(args[2], 'ignore'): + if len(args) == 3 and not self.readRegex(args[2], 'ignore'): # pragma: no cover return False except RegexException as e: output( 'ERROR: %s' % e ) @@ -521,31 +597,39 @@ class Fail2banRegex(object): output( "Use log file : %s" % cmd_log ) output( "Use encoding : %s" % self._encoding ) test_lines = self.file_lines_gen(hdlr) - except IOError as e: + except IOError as e: # pragma: no cover output( e ) return False - elif cmd_log == "systemd-journal": # pragma: no cover - if not journal: + elif cmd_log.startswith("systemd-journal"): # pragma: no cover + if not FilterSystemd: output( "Error: systemd library not found. Exiting..." ) return False - myjournal = journal.Reader(converters={'__CURSOR': lambda x: x}) + output( "Use systemd journal" ) + output( "Use encoding : %s" % self._encoding ) + backend, beArgs = JailReader.extractOptions(cmd_log) + flt = FilterSystemd(None, **beArgs) + flt.setLogEncoding(self._encoding) + myjournal = flt.getJournalReader() journalmatch = self._journalmatch self.setDatePattern(None) if journalmatch: - try: - for element in journalmatch: - if element == "+": - myjournal.add_disjunction() - else: - myjournal.add_match(element) - except ValueError: - output( "Error: Invalid journalmatch: %s" % shortstr(" ".join(journalmatch)) ) - return False + flt.addJournalMatch(journalmatch) output( "Use journal match : %s" % " ".join(journalmatch) ) - test_lines = journal_lines_gen(myjournal) + test_lines = journal_lines_gen(flt, myjournal) else: - output( "Use single line : %s" % shortstr(cmd_log) ) - test_lines = [ cmd_log ] + # if single line parsing (without buffering) + if self._filter.getMaxLines() <= 1: + output( "Use single line : %s" % shortstr(cmd_log.replace("\n", r"\n")) ) + test_lines = [ cmd_log ] + else: # multi line parsing (with buffering) + test_lines = cmd_log.split("\n") + output( "Use multi line : %s line(s)" % len(test_lines) ) + for i, l in enumerate(test_lines): + if i >= 5: + output( "| ..." ); break + output( "| %2.2s: %s" % (i+1, shortstr(l)) ) + output( "`-" ) + output( "" ) self.process(test_lines) @@ -598,5 +682,5 @@ def exec_command_line(*args): logSys.addHandler(stdout) fail2banRegex = Fail2banRegex(opts) - if not fail2banRegex.start(opts, args): + if not fail2banRegex.start(args): sys.exit(-1) diff --git a/fail2ban/client/filterreader.py b/fail2ban/client/filterreader.py index cdf0d8af..9edeb2f3 100644 --- a/fail2ban/client/filterreader.py +++ b/fail2ban/client/filterreader.py @@ -27,8 +27,7 @@ __license__ = "GPL" import os import shlex -from .configreader import DefinitionInitConfigReader, _merge_dicts -from ..server.action import CommandAction +from .configreader import DefinitionInitConfigReader from ..helpers import getLogger # Gets the instance of the logger. @@ -38,6 +37,7 @@ logSys = getLogger(__name__) class FilterReader(DefinitionInitConfigReader): _configOpts = { + "prefregex": ["string", None], "ignoreregex": ["string", None], "failregex": ["string", ""], "maxlines": ["int", None], @@ -52,17 +52,6 @@ class FilterReader(DefinitionInitConfigReader): def getFile(self): return self.__file - def getCombined(self): - combinedopts = self._opts - if self._initOpts: - combinedopts = _merge_dicts(self._opts, self._initOpts) - if not len(combinedopts): - return {} - opts = CommandAction.substituteRecursiveTags(combinedopts) - if not opts: - raise ValueError('recursive tag definitions unable to be resolved') - return opts - def convert(self): stream = list() opts = self.getCombined() @@ -70,6 +59,7 @@ class FilterReader(DefinitionInitConfigReader): return stream for opt, value in opts.iteritems(): if opt in ("failregex", "ignoreregex"): + if value is None: continue multi = [] for regex in value.split('\n'): # Do not send a command if the rule is empty. @@ -79,14 +69,14 @@ class FilterReader(DefinitionInitConfigReader): stream.append(["multi-set", self._jailName, "add" + opt, multi]) elif len(multi): stream.append(["set", self._jailName, "add" + opt, multi[0]]) - elif opt == 'maxlines': - # We warn when multiline regex is used without maxlines > 1 - # therefore keep sure we set this option first. - stream.insert(0, ["set", self._jailName, "maxlines", value]) - elif opt == 'datepattern': - stream.append(["set", self._jailName, "datepattern", value]) + elif opt in ('maxlines', 'prefregex'): + # Be sure we set this options first. + stream.insert(0, ["set", self._jailName, opt, value]) + elif opt in ('datepattern'): + stream.append(["set", self._jailName, opt, value]) # Do not send a command if the match is empty. elif opt == 'journalmatch': + if value is None: continue for match in value.split("\n"): if match == '': continue stream.append( diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index b223532e..ce0ed3b6 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -43,7 +43,7 @@ logSys = getLogger(__name__) class JailReader(ConfigReader): # regex, to extract list of options: - optionCRE = re.compile(r"^([\w\-_\.]+)(?:\[(.*)\])?\s*$", re.DOTALL) + optionCRE = re.compile(r"^([^\[]+)(?:\[(.*)\])?\s*$", re.DOTALL) # regex, to iterate over single option in option list, syntax: # `action = act[p1="...", p2='...', p3=...]`, where the p3=... not contains `,` or ']' # since v0.10 separator extended with `]\s*[` for support of multiple option groups, syntax @@ -101,6 +101,7 @@ class JailReader(ConfigReader): ["string", "filter", ""]] opts = [["bool", "enabled", False], ["string", "logpath", None], + ["string", "logtimezone", None], ["string", "logencoding", None], ["string", "backend", "auto"], ["int", "maxretry", None], @@ -110,6 +111,7 @@ class JailReader(ConfigReader): ["string", "failregex", None], ["string", "ignoreregex", None], ["string", "ignorecommand", None], + ["bool", "ignoreself", None], ["string", "ignoreip", None], ["string", "filter", ""], ["string", "datepattern", None], @@ -136,13 +138,14 @@ class JailReader(ConfigReader): if not filterName: raise JailDefError("Invalid filter definition %r" % flt) self.__filter = FilterReader( - filterName, self.__name, filterOpt, share_config=self.share_config, basedir=self.getBaseDir()) + filterName, self.__name, filterOpt, + share_config=self.share_config, basedir=self.getBaseDir()) ret = self.__filter.read() - # merge options from filter as 'known/...': - self.__filter.getOptions(self.__opts) - ConfigReader.merge_section(self, self.__name, self.__filter.getCombined(), 'known/') if not ret: raise JailDefError("Unable to read the filter %r" % filterName) + # merge options from filter as 'known/...' (all options unfiltered): + self.__filter.getOptions(self.__opts, all=True) + ConfigReader.merge_section(self, self.__name, self.__filter.getCombined(), 'known/') else: self.__filter = None logSys.warning("No filter set for jail %s" % self.__name) @@ -219,8 +222,8 @@ class JailReader(ConfigReader): if self.__filter: stream.extend(self.__filter.convert()) for opt, value in self.__opts.iteritems(): - if opt == "logpath" and \ - not self.__opts.get('backend', None).startswith("systemd"): + if opt == "logpath": + if self.__opts.get('backend', None).startswith("systemd"): continue found_files = 0 for path in value.split("\n"): path = path.rsplit(" ", 1) diff --git a/fail2ban/helpers.py b/fail2ban/helpers.py index 4f4426b0..556ca173 100644 --- a/fail2ban/helpers.py +++ b/fail2ban/helpers.py @@ -169,6 +169,36 @@ def splitwords(s): return [] return filter(bool, map(str.strip, re.split('[ ,\n]+', s))) +if sys.version_info >= (3,5): + eval(compile(r'''if 1: + def _merge_dicts(x, y): + """Helper to merge dicts. + """ + if y: + return {**x, **y} + return x + + def _merge_copy_dicts(x, y): + """Helper to merge dicts to guarantee a copy result (r is never x). + """ + return {**x, **y} + ''', __file__, 'exec')) +else: + def _merge_dicts(x, y): + """Helper to merge dicts. + """ + r = x + if y: + r = x.copy() + r.update(y) + return r + def _merge_copy_dicts(x, y): + """Helper to merge dicts to guarantee a copy result (r is never x). + """ + r = x.copy() + if y: + r.update(y) + return r # # Following "uni_decode" function unified python independent any to string converting @@ -200,6 +230,113 @@ else: raise return uni_decode(x, enc, 'replace') + +# +# Following facilities used for safe recursive interpolation of +# tags (<tag>) in tagged options. +# + +# max tag replacement count: +MAX_TAG_REPLACE_COUNT = 10 + +# compiled RE for tag name (replacement name) +TAG_CRE = re.compile(r'<([^ <>]+)>') + +def substituteRecursiveTags(inptags, conditional='', + ignore=(), addrepl=None +): + """Sort out tag definitions within other tags. + Since v.0.9.2 supports embedded interpolation (see test cases for examples). + + so: becomes: + a = 3 a = 3 + b = <a>_3 b = 3_3 + + Parameters + ---------- + inptags : dict + Dictionary of tags(keys) and their values. + + Returns + ------- + dict + Dictionary of tags(keys) and their values, with tags + within the values recursively replaced. + """ + #logSys = getLogger("fail2ban") + tre_search = TAG_CRE.search + # copy return tags dict to prevent modifying of inptags: + tags = inptags.copy() + # init: + ignore = set(ignore) + done = set() + noRecRepl = hasattr(tags, "getRawItem") + # repeat substitution while embedded-recursive (repFlag is True) + while True: + repFlag = False + # substitute each value: + for tag in tags.iterkeys(): + # ignore escaped or already done (or in ignore list): + if tag in ignore or tag in done: continue + # ignore replacing callable items from calling map - should be converted on demand only (by get): + if noRecRepl and callable(tags.getRawItem(tag)): continue + value = orgval = str(tags[tag]) + # search and replace all tags within value, that can be interpolated using other tags: + m = tre_search(value) + refCounts = {} + #logSys.log(5, 'TAG: %s, value: %s' % (tag, value)) + while m: + # found replacement tag: + rtag = m.group(1) + # don't replace tags that should be currently ignored (pre-replacement): + if rtag in ignore: + m = tre_search(value, m.end()) + continue + #logSys.log(5, 'found: %s' % rtag) + if rtag == tag or refCounts.get(rtag, 1) > MAX_TAG_REPLACE_COUNT: + # recursive definitions are bad + #logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) ) + raise ValueError( + "properties contain self referencing definitions " + "and cannot be resolved, fail tag: %s, found: %s in %s, value: %s" % + (tag, rtag, refCounts, value)) + repl = None + if conditional: + repl = tags.get(rtag + '?' + conditional) + if repl is None: + repl = tags.get(rtag) + # try to find tag using additional replacement (callable): + if repl is None and addrepl is not None: + repl = addrepl(rtag) + if repl is None: + # Missing tags - just continue on searching after end of match + # Missing tags are ok - cInfo can contain aInfo elements like <HOST> and valid shell + # constructs like <STDIN>. + m = tre_search(value, m.end()) + continue + # if calling map - be sure we've string: + if noRecRepl: repl = str(repl) + value = value.replace('<%s>' % rtag, repl) + #logSys.log(5, 'value now: %s' % value) + # increment reference count: + refCounts[rtag] = refCounts.get(rtag, 0) + 1 + # the next match for replace: + m = tre_search(value, m.start()) + #logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value)) + # was substituted? + if orgval != value: + # check still contains any tag - should be repeated (possible embedded-recursive substitution): + if tre_search(value): + repFlag = True + tags[tag] = value + # no more sub tags (and no possible composite), add this tag to done set (just to be faster): + if '<' not in value: done.add(tag) + # stop interpolation, if no replacements anymore: + if not repFlag: + break + return tags + + class BgService(object): """Background servicing diff --git a/fail2ban/protocol.py b/fail2ban/protocol.py index d1c33d88..3625ec01 100644 --- a/fail2ban/protocol.py +++ b/fail2ban/protocol.py @@ -81,6 +81,7 @@ protocol = [ ["status <JAIL> [FLAVOR]", "gets the current status of <JAIL>, with optional flavor or extended info"], ['', "JAIL CONFIGURATION", ""], ["set <JAIL> idle on|off", "sets the idle state of <JAIL>"], +["set <JAIL> ignoreself true|false", "allows the ignoring of own IP addresses"], ["set <JAIL> addignoreip <IP>", "adds <IP> to the ignore list of <JAIL>"], ["set <JAIL> delignoreip <IP>", "removes <IP> from the ignore list of <JAIL>"], ["set <JAIL> addlogpath <FILE> ['tail']", "adds <FILE> to the monitoring list of <JAIL>, optionally starting at the 'tail' of the file (default 'head')."], @@ -117,6 +118,7 @@ protocol = [ ["get <JAIL> logpath", "gets the list of the monitored files for <JAIL>"], ["get <JAIL> logencoding", "gets the encoding of the log files for <JAIL>"], ["get <JAIL> journalmatch", "gets the journal filter match for <JAIL>"], +["get <JAIL> ignoreself", "gets the current value of the ignoring the own IP addresses"], ["get <JAIL> ignoreip", "gets the list of ignored IP addresses for <JAIL>"], ["get <JAIL> ignorecommand", "gets ignorecommand of <JAIL>"], ["get <JAIL> failregex", "gets the list of regular expressions which matches the failures for <JAIL>"], diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index 62aa51dc..d00458ba 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -32,10 +32,11 @@ import time from abc import ABCMeta from collections import MutableMapping +from .failregex import mapTag2Opt from .ipdns import asip from .mytime import MyTime from .utils import Utils -from ..helpers import getLogger +from ..helpers import getLogger, _merge_copy_dicts, substituteRecursiveTags, TAG_CRE, MAX_TAG_REPLACE_COUNT # Gets the instance of the logger. logSys = getLogger(__name__) @@ -46,14 +47,19 @@ _cmd_lock = threading.Lock() # Todo: make it configurable resp. automatically set, ex.: `[ -f /proc/net/if_inet6 ] && echo 'yes' || echo 'no'`: allowed_ipv6 = True -# max tag replacement count: -MAX_TAG_REPLACE_COUNT = 10 +# capture groups from filter for map to ticket data: +FCUSTAG_CRE = re.compile(r'<F-([A-Z0-9_\-]+)>'); # currently uppercase only -# compiled RE for tag name (replacement name) -TAG_CRE = re.compile(r'<([^ <>]+)>') +CONDITIONAL_FAM_RE = re.compile(r"^(\w+)\?(family)=") +# New line, space +ADD_REPL_TAGS = { + "br": "\n", + "sp": " " +} -class CallingMap(MutableMapping): + +class CallingMap(MutableMapping, object): """A Mapping type which returns the result of callable values. `CallingMap` behaves similar to a standard python dictionary, @@ -70,23 +76,71 @@ class CallingMap(MutableMapping): The dictionary data which can be accessed to obtain items uncalled """ + # immutable=True saves content between actions, without interim copying (save original on demand, recoverable via reset) + __slots__ = ('data', 'storage', 'immutable', '__org_data') def __init__(self, *args, **kwargs): + self.storage = dict() + self.immutable = True self.data = dict(*args, **kwargs) + def reset(self, immutable=True): + self.storage = dict() + try: + self.data = self.__org_data + except AttributeError: + pass + self.immutable = immutable + def __repr__(self): - return "%s(%r)" % (self.__class__.__name__, self.data) + return "%s(%r)" % (self.__class__.__name__, self._asdict()) + + def _asdict(self): + try: + return dict(self) + except: + return dict(self.data, **self.storage) + + def getRawItem(self, key): + try: + value = self.storage[key] + except KeyError: + value = self.data[key] + return value def __getitem__(self, key): - value = self.data[key] + try: + value = self.storage[key] + except KeyError: + value = self.data[key] if callable(value): - return value() - else: - return value + # check arguments can be supplied to callable (for backwards compatibility): + value = value(self) if hasattr(value, '__code__') and value.__code__.co_argcount else value() + self.storage[key] = value + return value def __setitem__(self, key, value): - self.data[key] = value + # mutate to copy: + if self.immutable: + self.storage = self.storage.copy() + self.__org_data = self.data + self.data = self.data.copy() + self.immutable = False + self.storage[key] = value + + def __unavailable(self, key): + raise KeyError("Key %r was deleted" % key) def __delitem__(self, key): + # mutate to copy: + if self.immutable: + self.storage = self.storage.copy() + self.__org_data = self.data + self.data = self.data.copy() + self.immutable = False + try: + del self.storage[key] + except KeyError: + pass del self.data[key] def __iter__(self): @@ -95,8 +149,8 @@ class CallingMap(MutableMapping): def __len__(self): return len(self.data) - def copy(self): - return self.__class__(self.data.copy()) + def copy(self): # pargma: no cover + return self.__class__(_merge_copy_dicts(self.data, self.storage)) class ActionBase(object): @@ -149,17 +203,17 @@ class ActionBase(object): self._name = name self._logSys = getLogger("fail2ban.%s" % self.__class__.__name__) - def start(self): + def start(self): # pragma: no cover - abstract """Executed when the jail/action is started. """ pass - def stop(self): + def stop(self): # pragma: no cover - abstract """Executed when the jail/action is stopped. """ pass - def ban(self, aInfo): + def ban(self, aInfo): # pragma: no cover - abstract """Executed when a ban occurs. Parameters @@ -170,7 +224,7 @@ class ActionBase(object): """ pass - def unban(self, aInfo): + def unban(self, aInfo): # pragma: no cover - abstract """Executed when a ban expires. Parameters @@ -219,14 +273,16 @@ class CommandAction(ActionBase): self.timeout = 60 ## Command executed in order to initialize the system. self.actionstart = '' - ## Command executed when an IP address gets banned. + ## Command executed when ticket gets banned. self.actionban = '' - ## Command executed when an IP address gets removed. + ## Command executed when ticket gets removed. self.actionunban = '' ## Command executed in order to check requirements. self.actioncheck = '' ## Command executed in order to restore sane environment in error case. self.actionrepair = '' + ## Command executed in order to flush all bans at once (e. g. by stop/shutdown the system). + self.actionflush = '' ## Command executed in order to stop the system. self.actionstop = '' ## Command executed in case of reloading action. @@ -238,6 +294,7 @@ class CommandAction(ActionBase): super(CommandAction, self).__init__(jail, name) self.__init = 1 self.__properties = None + self.__started = {} self.__substCache = {} self.clearAllParams() self._logSys.debug("Created %s" % self.__class__) @@ -259,6 +316,16 @@ class CommandAction(ActionBase): # set: self.__dict__[name] = value + def __delattr__(self, name): + if not name.startswith('_'): + # parameters changed - clear properties and substitution cache: + self.__properties = None + self.__substCache.clear() + #self._logSys.debug("Unset action %r %s", self._name, name) + self._logSys.debug(" Unset %s", name) + # del: + del self.__dict__[name] + @property def _properties(self): """A dictionary of the actions properties. @@ -280,7 +347,11 @@ class CommandAction(ActionBase): def _substCache(self): return self.__substCache - def _executeOperation(self, tag, operation): + def _getOperation(self, tag, family): + return self.replaceTag(tag, self._properties, + conditional=('family=' + family), cache=self.__substCache) + + def _executeOperation(self, tag, operation, family=[]): """Executes the operation commands (like "actionstart", "actionstop", etc). Replace the tags in the action command with actions properties @@ -290,14 +361,14 @@ class CommandAction(ActionBase): res = True try: # common (resp. ipv4): - startCmd = self.replaceTag(tag, self._properties, - conditional='family=inet4', cache=self.__substCache) - if startCmd: - res &= self.executeCmd(startCmd, self.timeout) + startCmd = None + if not family or 'inet4' in family: + startCmd = self._getOperation(tag, 'inet4') + if startCmd: + res &= self.executeCmd(startCmd, self.timeout) # start ipv6 actions if available: - if allowed_ipv6: - startCmd6 = self.replaceTag(tag, self._properties, - conditional='family=inet6', cache=self.__substCache) + if allowed_ipv6 and (not family or 'inet6' in family): + startCmd6 = self._getOperation(tag, 'inet6') if startCmd6 and startCmd6 != startCmd: res &= self.executeCmd(startCmd6, self.timeout) if not res: @@ -305,13 +376,34 @@ class CommandAction(ActionBase): except ValueError as e: raise RuntimeError("Error %s action %s/%s: %r" % (operation, self._jail, self._name, e)) - def start(self): + COND_FAMILIES = {'inet4':1, 'inet6':1} + + @property + def _startOnDemand(self): + """Checks the action depends on family (conditional)""" + v = self._properties.get('actionstart_on_demand') + if v is None: + v = False + for n in self._properties: + if CONDITIONAL_FAM_RE.match(n): + v = True + break + self._properties['actionstart_on_demand'] = v + return v + + def start(self, family=[]): """Executes the "actionstart" command. Replace the tags in the action command with actions properties and executes the resulting command. """ - return self._executeOperation('<actionstart>', 'starting') + if not family: + # check the action depends on family (conditional): + if self._startOnDemand: + return True + elif self.__started.get(family): # pragma: no cover - normally unreachable + return True + return self._executeOperation('<actionstart>', 'starting', family=family) def ban(self, aInfo): """Executes the "actionban" command. @@ -325,6 +417,20 @@ class CommandAction(ActionBase): Dictionary which includes information in relation to the ban. """ + # if we should start the action on demand (conditional by family): + if self._startOnDemand: + family = aInfo.get('family') + if not self.__started.get(family): + self.start(family) + self.__started[family] = 1 + # mark also another families as "started" (-1), if they are equal + # (on demand, but the same for ipv4 and ipv6): + cmd = self._getOperation('<actionstart>', family) + for f in CommandAction.COND_FAMILIES: + if f != family and not self.__started.get(f): + if cmd == self._getOperation('<actionstart>', f): + self.__started[f] = -1 + # ban: if not self._processCmd('<actionban>', aInfo): raise RuntimeError("Error banning %(ip)s" % aInfo) @@ -343,13 +449,41 @@ class CommandAction(ActionBase): if not self._processCmd('<actionunban>', aInfo): raise RuntimeError("Error unbanning %(ip)s" % aInfo) + def flush(self): + """Executes the "actionflush" command. + + Command executed in order to flush all bans at once (e. g. by stop/shutdown + the system), instead of unbunning of each single ticket. + + Replaces the tags in the action command with actions properties + and executes the resulting command. + """ + family = [] + # cumulate started families, if started on demand (conditional): + if self._startOnDemand: + for f in CommandAction.COND_FAMILIES: + if self.__started.get(f) == 1: # only real started: + family.append(f) + # if no started (on demand) actions: + if not family: return True + return self._executeOperation('<actionflush>', 'flushing', family=family) + def stop(self): """Executes the "actionstop" command. Replaces the tags in the action command with actions properties and executes the resulting command. """ - return self._executeOperation('<actionstop>', 'stopping') + family = [] + # cumulate started families, if started on demand (conditional): + if self._startOnDemand: + for f in CommandAction.COND_FAMILIES: + if self.__started.get(f) == 1: # only real started: + family.append(f) + self.__started[f] = 0 + # if no started (on demand) actions: + if not family: return True + return self._executeOperation('<actionstop>', 'stopping', family=family) def reload(self, **kwargs): """Executes the "actionreload" command. @@ -364,83 +498,6 @@ class CommandAction(ActionBase): """ return self._executeOperation('<actionreload>', 'reloading') - @classmethod - def substituteRecursiveTags(cls, inptags, conditional=''): - """Sort out tag definitions within other tags. - Since v.0.9.2 supports embedded interpolation (see test cases for examples). - - so: becomes: - a = 3 a = 3 - b = <a>_3 b = 3_3 - - Parameters - ---------- - inptags : dict - Dictionary of tags(keys) and their values. - - Returns - ------- - dict - Dictionary of tags(keys) and their values, with tags - within the values recursively replaced. - """ - # copy return tags dict to prevent modifying of inptags: - tags = inptags.copy() - t = TAG_CRE - # repeat substitution while embedded-recursive (repFlag is True) - done = cls._escapedTags.copy() - while True: - repFlag = False - # substitute each value: - for tag in tags.iterkeys(): - # ignore escaped or already done: - if tag in done: continue - value = str(tags[tag]) - # search and replace all tags within value, that can be interpolated using other tags: - m = t.search(value) - refCounts = {} - #logSys.log(5, 'TAG: %s, value: %s' % (tag, value)) - while m: - found_tag = m.group(1) - #logSys.log(5, 'found: %s' % found_tag) - if found_tag == tag or refCounts.get(found_tag, 1) > MAX_TAG_REPLACE_COUNT: - # recursive definitions are bad - #logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) ) - raise ValueError( - "properties contain self referencing definitions " - "and cannot be resolved, fail tag: %s, found: %s in %s, value: %s" % - (tag, found_tag, refCounts, value)) - repl = None - if found_tag not in cls._escapedTags: - repl = tags.get(found_tag + '?' + conditional) - if repl is None: - repl = tags.get(found_tag) - if repl is None: - # Escaped or missing tags - just continue on searching after end of match - # Missing tags are ok - cInfo can contain aInfo elements like <HOST> and valid shell - # constructs like <STDIN>. - m = t.search(value, m.end()) - continue - value = value.replace('<%s>' % found_tag, repl) - #logSys.log(5, 'value now: %s' % value) - # increment reference count: - refCounts[found_tag] = refCounts.get(found_tag, 0) + 1 - # the next match for replace: - m = t.search(value, m.start()) - #logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value)) - # was substituted? - if tags[tag] != value: - # check still contains any tag - should be repeated (possible embedded-recursive substitution): - if t.search(value): - repFlag = True - tags[tag] = value - # no more sub tags (and no possible composite), add this tag to done set (just to be faster): - if '<' not in value: done.add(tag) - # stop interpolation, if no replacements anymore: - if not repFlag: - break - return tags - @staticmethod def escapeTag(value): """Escape characters which may be used for command injection. @@ -483,33 +540,150 @@ class CommandAction(ActionBase): str `query` string with tags replaced. """ + if '<' not in query: return query + # use cache if allowed: if cache is not None: ckey = (query, conditional) - string = cache.get(ckey) - if string is not None: - return string - # replace: - string = query - aInfo = cls.substituteRecursiveTags(aInfo, conditional) - for tag in aInfo: - if "<%s>" % tag in query: - value = aInfo.get(tag + '?' + conditional) + try: + return cache[ckey] + except KeyError: + pass + + # **Important**: don't replace if calling map - contains dynamic values only, + # no recursive tags, otherwise may be vulnerable on foreign user-input: + noRecRepl = isinstance(aInfo, CallingMap) + subInfo = aInfo + if not noRecRepl: + # substitute tags recursive (and cache if possible), + # first try get cached tags dictionary: + subInfo = csubkey = None + if cache is not None: + csubkey = ('subst-tags', id(aInfo), conditional) + try: + subInfo = cache[csubkey] + except KeyError: + pass + # interpolation of dictionary: + if subInfo is None: + subInfo = substituteRecursiveTags(aInfo, conditional, ignore=cls._escapedTags) + # cache if possible: + if csubkey is not None: + cache[csubkey] = subInfo + + # substitution callable, used by interpolation of each tag + def substVal(m): + tag = m.group(1) # tagname from match + value = None + if conditional: + value = subInfo.get(tag + '?' + conditional) + if value is None: + value = subInfo.get(tag) if value is None: - value = aInfo.get(tag) - value = str(value) # assure string - if tag in cls._escapedTags: - # That one needs to be escaped since its content is - # out of our control - value = cls.escapeTag(value) - string = string.replace('<' + tag + '>', value) - # New line, space - string = reduce(lambda s, kv: s.replace(*kv), (("<br>", '\n'), ("<sp>", " ")), string) - # cache if properties: + # fallback (no or default replacement) + return ADD_REPL_TAGS.get(tag, m.group()) + value = str(value) # assure string + if tag in cls._escapedTags: + # That one needs to be escaped since its content is + # out of our control + value = cls.escapeTag(value) + # replacement for tag: + return value + + # interpolation of query: + count = MAX_TAG_REPLACE_COUNT + 1 + while True: + value = TAG_CRE.sub(substVal, query) + # **Important**: no recursive replacement for tags from calling map (properties only): + if noRecRepl: break + # possible recursion ? + if value == query or '<' not in value: break + query = value + count -= 1 + if count <= 0: + raise ValueError( + "unexpected too long replacement interpolation, " + "possible self referencing definitions in query: %s" % (query,)) + + # cache if possible: if cache is not None: - cache[ckey] = string + cache[ckey] = value # - return string + return value + + ESCAPE_CRE = re.compile(r"""[\\#&;`|*?~<>\^\(\)\[\]{}$'"\n\r]""") + ESCAPE_VN_CRE = re.compile(r"\W") + + @classmethod + def replaceDynamicTags(cls, realCmd, aInfo): + """Replaces dynamical tags in `query` with property values. + + **Important** + ------------- + Because this tags are dynamic resp. foreign (user) input: + - values should be escaped (using "escape" as shell variable) + - no recursive substitution (no interpolation for <a<b>>) + - don't use cache + + Parameters + ---------- + query : str + String with tags. + aInfo : dict + Tags(keys) and associated values for substitution in query. + + Returns + ------- + str + shell script as string or array with tags replaced (direct or as variables). + """ + # array for escaped vars: + varsDict = dict() + + def escapeVal(tag, value): + # if the value should be escaped: + if cls.ESCAPE_CRE.search(value): + # That one needs to be escaped since its content is + # out of our control + tag = 'f2bV_%s' % cls.ESCAPE_VN_CRE.sub('_', tag) + varsDict[tag] = value # add variable + value = '$'+tag # replacement as variable + # replacement for tag: + return value + + # substitution callable, used by interpolation of each tag + def substVal(m): + tag = m.group(1) # tagname from match + try: + value = aInfo[tag] + except KeyError: + # fallback (no or default replacement) + return ADD_REPL_TAGS.get(tag, m.group()) + value = str(value) # assure string + # replacement for tag: + return escapeVal(tag, value) + + # Replace normally properties of aInfo non-recursive: + realCmd = TAG_CRE.sub(substVal, realCmd) + + # Replace ticket options (filter capture groups) non-recursive: + if '<' in realCmd: + tickData = aInfo.get("F-*") + if not tickData: tickData = {} + def substTag(m): + tag = mapTag2Opt(m.groups()[0]) + try: + value = str(tickData[tag]) + except KeyError: + return "" + return escapeVal("F_"+tag, value) + + realCmd = FCUSTAG_CRE.sub(substTag, realCmd) + + # build command corresponding "escaped" variables: + if varsDict: + realCmd = Utils.buildShellCmd(realCmd, varsDict) + return realCmd def _processCmd(self, cmd, aInfo=None, conditional=''): """Executes a command with preliminary checks and substitutions. @@ -575,9 +749,9 @@ class CommandAction(ActionBase): realCmd = self.replaceTag(cmd, self._properties, conditional=conditional, cache=self.__substCache) - # Replace tags + # Replace dynamical tags, important - don't cache, no recursion and auto-escape here if aInfo is not None: - realCmd = self.replaceTag(realCmd, aInfo, conditional=conditional) + realCmd = self.replaceDynamicTags(realCmd, aInfo) else: realCmd = cmd @@ -611,8 +785,5 @@ class CommandAction(ActionBase): logSys.debug("Nothing to do") return True - _cmd_lock.acquire() - try: + with _cmd_lock: return Utils.executeCmd(realCmd, timeout, shell=True, output=False, **kwargs) - finally: - _cmd_lock.release() diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index dd4c97c2..f8f8d4e1 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -35,6 +35,7 @@ except ImportError: OrderedDict = dict from .banmanager import BanManager +from .ipdns import DNSUtils from .jailthread import JailThread from .action import ActionBase, CommandAction, CallingMap from .mytime import MyTime @@ -150,7 +151,7 @@ class Actions(JailThread, Mapping): # reload actions after all parameters set via stream: for name, initOpts in self._reload_actions.iteritems(): if name in self._actions: - self._actions[name].reload(**initOpts if initOpts else {}) + self._actions[name].reload(**(initOpts if initOpts else {})) # remove obsolete actions (untouched by reload process): delacts = OrderedDict((name, action) for name, action in self._actions.iteritems() if name not in self._reload_actions) @@ -286,44 +287,91 @@ class Actions(JailThread, Mapping): self.stopActions() return True - def __getBansMerged(self, mi, overalljails=False): - """Gets bans merged once, a helper for lambda(s), prevents stop of executing action by any exception inside. + class ActionInfo(CallingMap): + + AI_DICT = { + "ip": lambda self: self.__ticket.getIP(), + "family": lambda self: self['ip'].familyStr, + "ip-rev": lambda self: self['ip'].getPTR(''), + "ip-host": lambda self: self['ip'].getHost(), + "fid": lambda self: self.__ticket.getID(), + "failures": lambda self: self.__ticket.getAttempt(), + "time": lambda self: self.__ticket.getTime(), + "matches": lambda self: "\n".join(self.__ticket.getMatches()), + # to bypass actions, that should not be executed for restored tickets + "restored": lambda self: (1 if self.__ticket.restored else 0), + # extra-interpolation - all match-tags (captured from the filter): + "F-*": lambda self, tag=None: self.__ticket.getData(tag), + # merged info: + "ipmatches": lambda self: "\n".join(self._mi4ip(True).getMatches()), + "ipjailmatches": lambda self: "\n".join(self._mi4ip().getMatches()), + "ipfailures": lambda self: self._mi4ip(True).getAttempt(), + "ipjailfailures": lambda self: self._mi4ip().getAttempt(), + # system-information: + "fq-hostname": lambda self: DNSUtils.getHostname(fqdn=True), + "sh-hostname": lambda self: DNSUtils.getHostname(fqdn=False) + } + + __slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip') + + def __init__(self, ticket, jail=None, immutable=True, data=AI_DICT): + self.__ticket = ticket + self.__jail = jail + self.storage = dict() + self.immutable = immutable + self.data = data + + def copy(self): # pargma: no cover + return self.__class__(self.__ticket, self.__jail, self.immutable, self.data.copy()) + + def _mi4ip(self, overalljails=False): + """Gets bans merged once, a helper for lambda(s), prevents stop of executing action by any exception inside. + + This function never returns None for ainfo lambdas - always a ticket (merged or single one) + and prevents any errors through merging (to guarantee ban actions will be executed). + [TODO] move merging to observer - here we could wait for merge and read already merged info from a database + + Parameters + ---------- + overalljails : bool + switch to get a merged bans : + False - (default) bans merged for current jail only + True - bans merged for all jails of current ip address + + Returns + ------- + BanTicket + merged or self ticket only + """ + if not hasattr(self, '__mi4ip'): + self.__mi4ip = {} + mi = self.__mi4ip + idx = 'all' if overalljails else 'jail' + if idx in mi: + return mi[idx] if mi[idx] is not None else self.__ticket + try: + jail = self.__jail + ip = self['ip'] + mi[idx] = None + if not jail.database: # pragma: no cover + return self.__ticket + if overalljails: + mi[idx] = jail.database.getBansMerged(ip=ip) + else: + mi[idx] = jail.database.getBansMerged(ip=ip, jail=jail) + except Exception as e: + logSys.error( + "Failed to get %s bans merged, jail '%s': %s", + idx, jail.name, e, + exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) + return mi[idx] if mi[idx] is not None else self.__ticket - This function never returns None for ainfo lambdas - always a ticket (merged or single one) - and prevents any errors through merging (to guarantee ban actions will be executed). - [TODO] move merging to observer - here we could wait for merge and read already merged info from a database - Parameters - ---------- - mi : dict - merge info, initial for lambda should contains {ip, ticket} - overalljails : bool - switch to get a merged bans : - False - (default) bans merged for current jail only - True - bans merged for all jails of current ip address + def __getActionInfo(self, ticket): + ip = ticket.getIP() + aInfo = Actions.ActionInfo(ticket, self._jail) + return aInfo - Returns - ------- - BanTicket - merged or self ticket only - """ - idx = 'all' if overalljails else 'jail' - if idx in mi: - return mi[idx] if mi[idx] is not None else mi['ticket'] - try: - jail=self._jail - ip=mi['ip'] - mi[idx] = None - if overalljails: - mi[idx] = jail.database.getBansMerged(ip=ip) - else: - mi[idx] = jail.database.getBansMerged(ip=ip, jail=jail) - except Exception as e: - logSys.error( - "Failed to get %s bans merged, jail '%s': %s", - idx, jail.name, e, - exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) - return mi[idx] if mi[idx] is not None else mi['ticket'] def __checkBan(self): """Check for IP address to ban. @@ -341,27 +389,19 @@ class Actions(JailThread, Mapping): ticket = self._jail.getFailTicket() if not ticket: break - aInfo = CallingMap() bTicket = BanManager.createBanTicket(ticket) ip = bTicket.getIP() - aInfo["ip"] = ip - aInfo["failures"] = bTicket.getAttempt() - aInfo["time"] = bTicket.getTime() - aInfo["matches"] = "\n".join(bTicket.getMatches()) - if self._jail.database is not None: - mi4ip = lambda overalljails=False, self=self, \ - mi={'ip':ip, 'ticket':bTicket}: self.__getBansMerged(mi, overalljails) - aInfo["ipmatches"] = lambda: "\n".join(mi4ip(True).getMatches()) - aInfo["ipjailmatches"] = lambda: "\n".join(mi4ip().getMatches()) - aInfo["ipfailures"] = lambda: mi4ip(True).getAttempt() - aInfo["ipjailfailures"] = lambda: mi4ip().getAttempt() + aInfo = self.__getActionInfo(bTicket) reason = {} if self.__banManager.addBanTicket(bTicket, reason=reason): cnt += 1 logSys.notice("[%s] %sBan %s", self._jail.name, ('' if not bTicket.restored else 'Restore '), ip) for name, action in self._actions.iteritems(): try: - action.ban(aInfo.copy()) + if ticket.restored and getattr(action, 'norestored', False): + continue + if not aInfo.immutable: aInfo.reset() + action.ban(aInfo) except Exception as e: logSys.error( "Failed to execute ban jail '%s' action '%s' " @@ -411,25 +451,37 @@ class Actions(JailThread, Mapping): If actions specified, don't flush list - just execute unban for given actions (reload, obsolete resp. removed actions). """ + log = True if actions is None: logSys.debug("Flush ban list") lst = self.__banManager.flushBanList() else: + log = False # don't log "[jail] Unban ..." if removing actions only. lst = iter(self.__banManager) cnt = 0 + # first we'll execute flush for actions supporting this operation: + unbactions = {} + for name, action in (actions if actions is not None else self._actions).iteritems(): + if hasattr(action, 'flush') and action.actionflush: + logSys.notice("[%s] Flush ticket(s) with %s", self._jail.name, name) + action.flush() + else: + unbactions[name] = action + actions = unbactions + # unban each ticket with non-flasheable actions: for ticket in lst: # delete ip from database also: if db and self._jail.database is not None: ip = str(ticket.getIP()) self._jail.database.delBan(self._jail, ip) # unban ip: - self.__unBan(ticket, actions=actions) + self.__unBan(ticket, actions=actions, log=log) cnt += 1 logSys.debug("Unbanned %s, %s ticket(s) in %r", cnt, self.__banManager.size(), self._jail.name) return cnt - def __unBan(self, ticket, actions=None): + def __unBan(self, ticket, actions=None, log=True): """Unbans host corresponding to the ticket. Executes the actions in order to unban the host given in the @@ -444,17 +496,17 @@ class Actions(JailThread, Mapping): unbactions = self._actions else: unbactions = actions - aInfo = dict() - aInfo["ip"] = ticket.getIP() - aInfo["failures"] = ticket.getAttempt() - aInfo["time"] = ticket.getTime() - aInfo["matches"] = "".join(ticket.getMatches()) - if actions is None: + ip = ticket.getIP() + aInfo = self.__getActionInfo(ticket) + if log: logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"]) for name, action in unbactions.iteritems(): try: - logSys.debug("[%s] action %r: unban %s", self._jail.name, name, aInfo["ip"]) - action.unban(aInfo.copy()) + if ticket.restored and getattr(action, 'norestored', False): + continue + logSys.debug("[%s] action %r: unban %s", self._jail.name, name, ip) + if not aInfo.immutable: aInfo.reset() + action.unban(aInfo) except Exception as e: logSys.error( "Failed to execute unban jail '%s' action '%s' " diff --git a/fail2ban/server/database.py b/fail2ban/server/database.py index 77786f57..f4f9b6c2 100644 --- a/fail2ban/server/database.py +++ b/fail2ban/server/database.py @@ -593,7 +593,7 @@ class Fail2BanDb(object): if ip is not None: query += " AND ip=?" queryArgs.append(ip) - if forbantime is not None: + if forbantime not in (None, -1): # not specified or persistent (all) query += " AND timeofban > ?" queryArgs.append(fromtime - forbantime) if ip is None: diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index cccbf71a..90cfe1fd 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -27,6 +27,7 @@ import time from threading import Lock from .datetemplate import re, DateTemplate, DatePatternRegex, DateTai64n, DateEpoch +from .strptime import validateTimeZone from .utils import Utils from ..helpers import getLogger @@ -222,6 +223,8 @@ class DateDetector(object): self.__firstUnused = 0 # pre-match pattern: self.__preMatch = None + # default TZ (if set, treat log lines without explicit time zone to be in this time zone): + self.__default_tz = None def _appendTemplate(self, template, ignoreDup=False): name = template.name @@ -284,7 +287,7 @@ class DateDetector(object): if preMatch is not None: # get cached or create a copy with modified name/pattern, using preMatch replacement for {DATE}: template = _getAnchoredTemplate(template, - wrap=lambda s: RE_DATE_PREMATCH.sub(s, preMatch)) + wrap=lambda s: RE_DATE_PREMATCH.sub(lambda m: s, preMatch)) # append date detector template (ignore duplicate if some was added before default): self._appendTemplate(template, ignoreDup=ignoreDup) @@ -423,6 +426,14 @@ class DateDetector(object): logSys.log(logLevel, " no template.") return (None, None) + @property + def default_tz(self): + return self.__default_tz + + @default_tz.setter + def default_tz(self, value): + self.__default_tz = validateTimeZone(value) + def getTime(self, line, timeMatch=None): """Attempts to return the date on a log line using templates. @@ -449,7 +460,7 @@ class DateDetector(object): template = timeMatch[1] if template is not None: try: - date = template.getDate(line, timeMatch[0]) + date = template.getDate(line, timeMatch[0], default_tz=self.__default_tz) if date is not None: if logSys.getEffectiveLevel() <= logLevel: # pragma: no cover - heavy debug logSys.log(logLevel, " got time %f for %r using template %s", diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index 1d0b014b..44ea54a8 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -158,7 +158,7 @@ class DateTemplate(object): return dateMatch @abstractmethod - def getDate(self, line, dateMatch=None): + def getDate(self, line, dateMatch=None, default_tz=None): """Abstract method, which should return the date for a log line This should return the date for a log line, typically taking the @@ -169,6 +169,8 @@ class DateTemplate(object): ---------- line : str Log line, of which the date should be extracted from. + default_tz: if no explicit time zone is present in the line + passing this will interpret it as in that time zone. Raises ------ @@ -200,13 +202,14 @@ class DateEpoch(DateTemplate): regex = r"((?P<square>(?<=^\[))?\d{10,11}\b(?:\.\d{3,6})?)(?(square)(?=\]))" self.setRegex(regex, wordBegin='start', wordEnd=True) - def getDate(self, line, dateMatch=None): + def getDate(self, line, dateMatch=None, default_tz=None): """Method to return the date for a log line. Parameters ---------- line : str Log line, of which the date should be extracted from. + default_tz: ignored, Unix timestamps are time zone independent Returns ------- @@ -277,7 +280,7 @@ class DatePatternRegex(DateTemplate): regex = r'(?iu)' + regex super(DatePatternRegex, self).setRegex(regex, wordBegin, wordEnd) - def getDate(self, line, dateMatch=None): + def getDate(self, line, dateMatch=None, default_tz=None): """Method to return the date for a log line. This uses a custom version of strptime, using the named groups @@ -287,6 +290,7 @@ class DatePatternRegex(DateTemplate): ---------- line : str Log line, of which the date should be extracted from. + default_tz: optionally used to correct timezone Returns ------- @@ -297,7 +301,8 @@ class DatePatternRegex(DateTemplate): if not dateMatch: dateMatch = self.matchDate(line) if dateMatch: - return reGroupDictStrptime(dateMatch.groupdict()), dateMatch + return (reGroupDictStrptime(dateMatch.groupdict(), default_tz=default_tz), + dateMatch) class DateTai64n(DateTemplate): @@ -315,13 +320,14 @@ class DateTai64n(DateTemplate): # We already know the format for TAI64N self.setRegex("@[0-9a-f]{24}", wordBegin=wordBegin) - def getDate(self, line, dateMatch=None): + def getDate(self, line, dateMatch=None, default_tz=None): """Method to return the date for a log line. Parameters ---------- line : str Log line, of which the date should be extracted from. + default_tz: ignored, since TAI is time zone independent Returns ------- diff --git a/fail2ban/server/failregex.py b/fail2ban/server/failregex.py index 7c51ddb8..d5c9345f 100644 --- a/fail2ban/server/failregex.py +++ b/fail2ban/server/failregex.py @@ -27,6 +27,68 @@ import sys from .ipdns import IPAddr + +FTAG_CRE = re.compile(r'</?[\w\-]+/?>') + +FCUSTNAME_CRE = re.compile(r'^(/?)F-([A-Z0-9_\-]+)$'); # currently uppercase only + +R_HOST = [ + # separated ipv4: + r"""(?:::f{4,6}:)?(?P<ip4>%s)""" % (IPAddr.IP_4_RE,), + # separated ipv6: + r"""(?P<ip6>%s)""" % (IPAddr.IP_6_RE,), + # place-holder for ipv6 enclosed in optional [] (used in addr-, host-regex) + "", + # separated dns: + r"""(?P<dns>[\w\-.^_]*\w)""", + # place-holder for ADDR tag-replacement (joined): + "", + # place-holder for HOST tag replacement (joined): + "" +] +RI_IPV4 = 0 +RI_IPV6 = 1 +RI_IPV6BR = 2 +RI_DNS = 3 +RI_ADDR = 4 +RI_HOST = 5 + +R_HOST[RI_IPV6BR] = r"""\[?%s\]?""" % (R_HOST[RI_IPV6],) +R_HOST[RI_ADDR] = "(?:%s)" % ("|".join((R_HOST[RI_IPV4], R_HOST[RI_IPV6BR])),) +R_HOST[RI_HOST] = "(?:%s)" % ("|".join((R_HOST[RI_IPV4], R_HOST[RI_IPV6BR], R_HOST[RI_DNS])),) + +RH4TAG = { + # separated ipv4 (self closed, closed): + "IP4": R_HOST[RI_IPV4], + "F-IP4/": R_HOST[RI_IPV4], + # separated ipv6 (self closed, closed): + "IP6": R_HOST[RI_IPV6], + "F-IP6/": R_HOST[RI_IPV6], + # 2 address groups instead of <ADDR> - in opposition to `<HOST>`, + # for separate usage of 2 address groups only (regardless of `usedns`), `ip4` and `ip6` together + "ADDR": R_HOST[RI_ADDR], + "F-ADDR/": R_HOST[RI_ADDR], + # separated dns (self closed, closed): + "DNS": R_HOST[RI_DNS], + "F-DNS/": R_HOST[RI_DNS], + # default failure-id as no space tag: + "F-ID/": r"""(?P<fid>\S+)""", + # default failure port, like 80 or http : + "F-PORT/": r"""(?P<fport>\w+)""", +} + +# default failure groups map for customizable expressions (with different group-id): +R_MAP = { + "ID": "fid", + "PORT": "fport", +} + +def mapTag2Opt(tag): + try: # if should be mapped: + return R_MAP[tag] + except KeyError: + return tag.lower() + ## # Regular expression class. # @@ -41,20 +103,16 @@ class Regex: # avoid construction of invalid object. # @param value the regular expression - def __init__(self, regex, **kwargs): + def __init__(self, regex, multiline=False, **kwargs): self._matchCache = None # Perform shortcuts expansions. - # Resolve "<HOST>" tag using default regular expression for host: + # Replace standard f2b-tags (like "<HOST>", etc) using default regular expressions: regex = Regex._resolveHostTag(regex, **kwargs) - # Replace "<SKIPLINES>" with regular expression for multiple lines. - regexSplit = regex.split("<SKIPLINES>") - regex = regexSplit[0] - for n, regexLine in enumerate(regexSplit[1:]): - regex += "\n(?P<skiplines%i>(?:(.*\n)*?))" % n + regexLine + # if regex.lstrip() == '': raise RegexException("Cannot add empty regex") try: - self._regexObj = re.compile(regex, re.MULTILINE) + self._regexObj = re.compile(regex, re.MULTILINE if multiline else 0) self._regex = regex except sre_constants.error: raise RegexException("Unable to compile regular expression '%s'" % @@ -71,38 +129,52 @@ class Regex: @staticmethod def _resolveHostTag(regex, useDns="yes"): - # separated ipv4: - r_host = [] - r = r"""(?:::f{4,6}:)?(?P<ip4>%s)""" % (IPAddr.IP_4_RE,) - regex = regex.replace("<IP4>", r); # self closed - regex = regex.replace("<F-IP4/>", r); # closed - r_host.append(r) - # separated ipv6: - r = r"""(?P<ip6>%s)""" % (IPAddr.IP_6_RE,) - regex = regex.replace("<IP6>", r); # self closed - regex = regex.replace("<F-IP6/>", r); # closed - r_host.append(r"""\[?%s\]?""" % (r,)); # enclose ipv6 in optional [] in host-regex - # 2 address groups instead of <ADDR> - in opposition to `<HOST>`, - # for separate usage of 2 address groups only (regardless of `usedns`), `ip4` and `ip6` together - regex = regex.replace("<ADDR>", "(?:%s)" % ("|".join(r_host),)) - # separated dns: - r = r"""(?P<dns>[\w\-.^_]*\w)""" - regex = regex.replace("<DNS>", r); # self closed - regex = regex.replace("<F-DNS/>", r); # closed - if useDns not in ("no",): - r_host.append(r) - # 3 groups instead of <HOST> - separated ipv4, ipv6 and host (dns) - regex = regex.replace("<HOST>", "(?:%s)" % ("|".join(r_host),)) - # default failure-id as no space tag: - regex = regex.replace("<F-ID/>", r"""(?P<fid>\S+)"""); # closed - # default failure port, like 80 or http : - regex = regex.replace("<F-PORT/>", r"""(?P<port>\w+)"""); # closed - # default failure groups (begin / end tag) for customizable expressions: - for o,r in (('IP4', 'ip4'), ('IP6', 'ip6'), ('DNS', 'dns'), ('ID', 'fid'), ('PORT', 'fport')): - regex = regex.replace("<F-%s>" % o, "(?P<%s>" % r); # open tag - regex = regex.replace("</F-%s>" % o, ")"); # close tag - - return regex + + openTags = dict() + props = { + 'nl': 0, # new lines counter by <SKIPLINES> tag; + } + # tag interpolation callable: + def substTag(m): + tag = m.group() + tn = tag[1:-1] + # 3 groups instead of <HOST> - separated ipv4, ipv6 and host (dns) + if tn == "HOST": + return R_HOST[RI_HOST if useDns not in ("no",) else RI_ADDR] + # replace "<SKIPLINES>" with regular expression for multiple lines (by buffering with maxlines) + if tn == "SKIPLINES": + nl = props['nl'] + props['nl'] = nl + 1 + return r"\n(?P<skiplines%i>(?:(?:.*\n)*?))" % (nl,) + # static replacement from RH4TAG: + try: + return RH4TAG[tn] + except KeyError: + pass + + # (begin / end tag) for customizable expressions, additionally used as + # user custom tags (match will be stored in ticket data, can be used in actions): + m = FCUSTNAME_CRE.match(tn) + if m: # match F-... + m = m.groups() + tn = m[1] + # close tag: + if m[0]: + # check it was already open: + if openTags.get(tn): + return ")" + return tag; # tag not opened, use original + # open tag: + openTags[tn] = 1 + # if should be mapped: + tn = mapTag2Opt(tn) + return "(?P<%s>" % (tn,) + + # original, no replacement: + return tag + + # substitute tags: + return FTAG_CRE.sub(substTag, regex) ## # Gets the regular expression. @@ -121,40 +193,45 @@ class Regex: # method of this object. # @param a list of tupples. The tupples are ( prematch, datematch, postdatematch ) - def search(self, tupleLines): + def search(self, tupleLines, orgLines=None): self._matchCache = self._regexObj.search( "\n".join("".join(value[::2]) for value in tupleLines) + "\n") - if self.hasMatched(): - # Find start of the first line where the match was found - try: - self._matchLineStart = self._matchCache.string.rindex( - "\n", 0, self._matchCache.start() +1 ) + 1 - except ValueError: - self._matchLineStart = 0 - # Find end of the last line where the match was found - try: - self._matchLineEnd = self._matchCache.string.index( - "\n", self._matchCache.end() - 1) + 1 - except ValueError: - self._matchLineEnd = len(self._matchCache.string) - - lineCount1 = self._matchCache.string.count( - "\n", 0, self._matchLineStart) - lineCount2 = self._matchCache.string.count( - "\n", 0, self._matchLineEnd) - self._matchedTupleLines = tupleLines[lineCount1:lineCount2] - self._unmatchedTupleLines = tupleLines[:lineCount1] - - n = 0 - for skippedLine in self.getSkippedLines(): - for m, matchedTupleLine in enumerate( - self._matchedTupleLines[n:]): - if "".join(matchedTupleLine[::2]) == skippedLine: - self._unmatchedTupleLines.append( - self._matchedTupleLines.pop(n+m)) - n += m - break - self._unmatchedTupleLines.extend(tupleLines[lineCount2:]) + if self._matchCache: + if orgLines is None: orgLines = tupleLines + # if single-line: + if len(orgLines) <= 1: + self._matchedTupleLines = orgLines + self._unmatchedTupleLines = [] + else: + # Find start of the first line where the match was found + try: + matchLineStart = self._matchCache.string.rindex( + "\n", 0, self._matchCache.start() +1 ) + 1 + except ValueError: + matchLineStart = 0 + # Find end of the last line where the match was found + try: + matchLineEnd = self._matchCache.string.index( + "\n", self._matchCache.end() - 1) + 1 + except ValueError: + matchLineEnd = len(self._matchCache.string) + + lineCount1 = self._matchCache.string.count( + "\n", 0, matchLineStart) + lineCount2 = self._matchCache.string.count( + "\n", 0, matchLineEnd) + self._matchedTupleLines = orgLines[lineCount1:lineCount2] + self._unmatchedTupleLines = orgLines[:lineCount1] + n = 0 + for skippedLine in self.getSkippedLines(): + for m, matchedTupleLine in enumerate( + self._matchedTupleLines[n:]): + if "".join(matchedTupleLine[::2]) == skippedLine: + self._unmatchedTupleLines.append( + self._matchedTupleLines.pop(n+m)) + n += m + break + self._unmatchedTupleLines.extend(orgLines[lineCount2:]) # Checks if the previous call to search() matched. # @@ -167,6 +244,13 @@ class Regex: return False ## + # Returns all matched groups. + # + + def getGroups(self): + return self._matchCache.groupdict() + + ## # Returns skipped lines. # # This returns skipped lines captured by the <SKIPLINES> tag. @@ -243,6 +327,10 @@ class RegexException(Exception): # FAILURE_ID_GROPS = ("fid", "ip4", "ip6", "dns") +# Additionally allows multi-line failure-id (used for wrapping e. g. conn-id to host) +# +FAILURE_ID_PRESENTS = FAILURE_ID_GROPS + ("mlfid",) + ## # Regular expression class. # @@ -257,21 +345,17 @@ class FailRegex(Regex): # avoid construction of invalid object. # @param value the regular expression - def __init__(self, regex, **kwargs): + def __init__(self, regex, prefRegex=None, **kwargs): # Initializes the parent. Regex.__init__(self, regex, **kwargs) # Check for group "dns", "ip4", "ip6", "fid" - if not [grp for grp in FAILURE_ID_GROPS if grp in self._regexObj.groupindex]: + if (not [grp for grp in FAILURE_ID_PRESENTS if grp in self._regexObj.groupindex] + and (prefRegex is None or + not [grp for grp in FAILURE_ID_PRESENTS if grp in prefRegex._regexObj.groupindex]) + ): raise RegexException("No failure-id group in '%s'" % self._regex) ## - # Returns all matched groups. - # - - def getGroups(self): - return self._matchCache.groupdict() - - ## # Returns the matched failure id. # # This corresponds to the pattern matched by the named group from given groups. diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 0540dedc..c4f29878 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -34,10 +34,11 @@ from .failmanager import FailManagerEmpty, FailManager from .ipdns import DNSUtils, IPAddr from .ticket import FailTicket from .jailthread import JailThread -from .datedetector import DateDetector +from .datedetector import DateDetector, validateTimeZone from .mytime import MyTime from .failregex import FailRegex, Regex, RegexException from .action import CommandAction +from .utils import Utils from ..helpers import getLogger, PREFER_ENC # Gets the instance of the logger. @@ -65,6 +66,8 @@ class Filter(JailThread): self.jail = jail ## The failures manager. self.failManager = FailManager() + ## Regular expression pre-filtering matching the failures. + self.__prefRegex = None ## The regular expression list matching the failures. self.__failRegex = list() ## The regular expression list with expressions to ignore. @@ -73,6 +76,8 @@ class Filter(JailThread): self.setUseDns(useDns) ## The amount of time to look back. self.__findTime = 600 + ## Ignore own IPs flag: + self.__ignoreSelf = True ## The ignore IP list. self.__ignoreIpList = [] ## Size of line buffer @@ -82,10 +87,14 @@ class Filter(JailThread): ## Store last time stamp, applicable for multi-line self.__lastTimeText = "" self.__lastDate = None + ## if set, treat log lines without explicit time zone to be in this time zone + self.__logtimezone = None ## External command self.__ignoreCommand = False ## Default or preferred encoding (to decode bytes from file or journal): self.__encoding = PREFER_ENC + ## Cache temporary holds failures info (used by multi-line for wrapping e. g. conn-id to host): + self.__mlfidCache = None ## Error counter (protected, so can be used in filter implementations) ## if it reached 100 (at once), run-cycle will go idle self._errors = 0 @@ -99,7 +108,7 @@ class Filter(JailThread): self.ticks = 0 self.dateDetector = DateDetector() - logSys.debug("Created %s" % self) + logSys.debug("Created %s", self) def __repr__(self): return "%s(%r)" % (self.__class__.__name__, self.jail) @@ -129,6 +138,23 @@ class Filter(JailThread): self.delLogPath(path) delattr(self, '_reload_logs') + @property + def mlfidCache(self): + if self.__mlfidCache: + return self.__mlfidCache + self.__mlfidCache = Utils.Cache(maxCount=100, maxTime=5*60) + return self.__mlfidCache + + @property + def prefRegex(self): + return self.__prefRegex + @prefRegex.setter + def prefRegex(self, value): + if value: + self.__prefRegex = Regex(value, useDns=self.__useDns) + else: + self.__prefRegex = None + ## # Add a regular expression which matches the failure. # @@ -137,13 +163,11 @@ class Filter(JailThread): # @param value the regular expression def addFailRegex(self, value): + multiLine = self.getMaxLines() > 1 try: - regex = FailRegex(value, useDns=self.__useDns) + regex = FailRegex(value, prefRegex=self.__prefRegex, multiline=multiLine, + useDns=self.__useDns) self.__failRegex.append(regex) - if "\n" in regex.getRegex() and not self.getMaxLines() > 1: - logSys.warning( - "Mutliline regex set for jail %r " - "but maxlines not greater than 1", self.jailName) except RegexException as e: logSys.error(e) raise e @@ -158,18 +182,15 @@ class Filter(JailThread): del self.__failRegex[index] except IndexError: logSys.error("Cannot remove regular expression. Index %d is not " - "valid" % index) + "valid", index) ## - # Get the regular expression which matches the failure. + # Get the regular expressions as list. # - # @return the regular expression + # @return the regular expression list def getFailRegex(self): - failRegex = list() - for regex in self.__failRegex: - failRegex.append(regex.getRegex()) - return failRegex + return [regex.getRegex() for regex in self.__failRegex] ## # Add the regular expression which matches the failure. @@ -196,7 +217,7 @@ class Filter(JailThread): del self.__ignoreRegex[index] except IndexError: logSys.error("Cannot remove regular expression. Index %d is not " - "valid" % index) + "valid", index) ## # Get the regular expression which matches the failure. @@ -219,9 +240,9 @@ class Filter(JailThread): value = value.lower() # must be a string by now if not (value in ('yes', 'warn', 'no', 'raw')): logSys.error("Incorrect value %r specified for usedns. " - "Using safe 'no'" % (value,)) + "Using safe 'no'", value) value = 'no' - logSys.debug("Setting usedns = %s for %s" % (value, self)) + logSys.debug("Setting usedns = %s for %s", value, self) self.__useDns = value ## @@ -263,6 +284,7 @@ class Filter(JailThread): return else: dd = DateDetector() + dd.default_tz = self.__logtimezone if not isinstance(pattern, (list, tuple)): pattern = filter(bool, map(str.strip, re.split('\n+', pattern))) for pattern in pattern: @@ -289,6 +311,24 @@ class Filter(JailThread): return None ## + # Set the log default time zone + # + # @param tz the symbolic timezone (for now fixed offset only: UTC[+-]HHMM) + + def setLogTimeZone(self, tz): + validateTimeZone(tz); # avoid setting of wrong value, but hold original + self.__logtimezone = tz + if self.dateDetector: self.dateDetector.default_tz = self.__logtimezone + + ## + # Get the log default timezone + # + # @return symbolic timezone (a string) + + def getLogTimeZone(self): + return self.__logtimezone + + ## # Set the maximum retry value. # # @param value the retry value @@ -334,7 +374,7 @@ class Filter(JailThread): encoding = PREFER_ENC codecs.lookup(encoding) # Raise LookupError if invalid codec self.__encoding = encoding - logSys.info(" encoding: %s" % encoding) + logSys.info(" encoding: %s", encoding) return encoding ## @@ -379,7 +419,7 @@ class Filter(JailThread): if not isinstance(ip, IPAddr): ip = IPAddr(ip) if self.inIgnoreIPList(ip): - logSys.warning('Requested to manually ban an ignored IP %s. User knows best. Proceeding to ban it.' % ip) + logSys.warning('Requested to manually ban an ignored IP %s. User knows best. Proceeding to ban it.', ip) unixTime = MyTime.time() self.failManager.addFailure(FailTicket(ip, unixTime), self.failManager.getMaxRetry()) @@ -395,6 +435,17 @@ class Filter(JailThread): return ip ## + # Ignore own IP/DNS. + # + @property + def ignoreSelf(self): + return self.__ignoreSelf + + @ignoreSelf.setter + def ignoreSelf(self, value): + self.__ignoreSelf = value + + ## # Add an IP/DNS to the ignore list. # # IP addresses in the ignore list are not taken into account @@ -423,7 +474,7 @@ class Filter(JailThread): def logIgnoreIp(self, ip, log_ignore, ignore_source="unknown source"): if log_ignore: - logSys.info("[%s] Ignore %s by %s" % (self.jailName, ip, ignore_source)) + logSys.info("[%s] Ignore %s by %s", self.jailName, ip, ignore_source) def getIgnoreIP(self): return self.__ignoreIpList @@ -439,6 +490,11 @@ class Filter(JailThread): def inIgnoreIPList(self, ip, log_ignore=False): if not isinstance(ip, IPAddr): ip = IPAddr(ip) + + # check own IPs should be ignored and 'ip' is self IP: + if self.__ignoreSelf and ip in DNSUtils.getSelfIPs(): + return True + for net in self.__ignoreIpList: # check if the IP is covered by ignore IP if ip.isInNet(net): @@ -447,7 +503,7 @@ class Filter(JailThread): if self.__ignoreCommand: command = CommandAction.replaceTag(self.__ignoreCommand, { 'ip': ip } ) - logSys.debug('ignore command: ' + command) + logSys.debug('ignore command: %s', command) ret, ret_ignore = CommandAction.executeCmd(command, success_codes=(0, 1)) ret_ignore = ret and ret_ignore == 0 self.logIgnoreIp(ip, log_ignore and ret_ignore, ignore_source="command") @@ -486,10 +542,7 @@ class Filter(JailThread): for element in self.processLine(line, date): ip = element[1] unixTime = element[2] - lines = element[3] - fail = {} - if len(element) > 4: - fail = element[4] + fail = element[3] logSys.debug("Processing line with time:%s and ip:%s", unixTime, ip) if self.inIgnoreIPList(ip, log_ignore=True): @@ -497,7 +550,7 @@ class Filter(JailThread): logSys.info( "[%s] Found %s - %s", self.jailName, ip, datetime.datetime.fromtimestamp(unixTime).strftime("%Y-%m-%d %H:%M:%S") ) - tick = FailTicket(ip, unixTime, lines, data=fail) + tick = FailTicket(ip, unixTime, data=fail) self.failManager.addFailure(tick) # reset (halve) error counter (successfully processed line): if self._errors: @@ -532,6 +585,34 @@ class Filter(JailThread): return ignoreRegexIndex return None + def _mergeFailure(self, mlfid, fail, failRegex): + mlfidFail = self.mlfidCache.get(mlfid) if self.__mlfidCache else None + # if multi-line failure id (connection id) known: + if mlfidFail: + mlfidGroups = mlfidFail[1] + # update - if not forget (disconnect/reset): + if not fail.get('mlfforget'): + mlfidGroups.update(fail) + else: + self.mlfidCache.unset(mlfid) # remove cached entry + # merge with previous info: + fail2 = mlfidGroups.copy() + fail2.update(fail) + if not fail.get('nofail'): # be sure we've correct current state + try: + del fail2['nofail'] + except KeyError: + pass + fail2["matches"] = fail.get("matches", []) + failRegex.getMatchedTupleLines() + fail = fail2 + elif not fail.get('mlfforget'): + mlfidFail = [self.__lastDate, fail] + self.mlfidCache.set(mlfid, mlfidFail) + if fail.get('nofail'): + fail["matches"] = failRegex.getMatchedTupleLines() + return fail + + ## # Finds the failure in a line given split into time and log parts. # @@ -564,7 +645,7 @@ class Filter(JailThread): dateTimeMatch = self.dateDetector.getTime(timeText, tupleLine[3]) if dateTimeMatch is None: - logSys.error("findFailure failed to parse timeText: " + timeText) + logSys.error("findFailure failed to parse timeText: %s", timeText) date = self.__lastDate else: @@ -582,77 +663,126 @@ class Filter(JailThread): date, MyTime.time(), self.getFindTime()) return failList - self.__lineBuffer = ( - self.__lineBuffer + [tupleLine[:3]])[-self.__lineBufferSize:] - logSys.log(5, "Looking for failregex match of %r" % self.__lineBuffer) + if self.__lineBufferSize > 1: + orgBuffer = self.__lineBuffer = ( + self.__lineBuffer + [tupleLine[:3]])[-self.__lineBufferSize:] + else: + orgBuffer = self.__lineBuffer = [tupleLine[:3]] + logSys.log(5, "Looking for match of %r", self.__lineBuffer) + + # Pre-filter fail regex (if available): + preGroups = {} + if self.__prefRegex: + if logSys.getEffectiveLevel() <= logging.HEAVYDEBUG: # pragma: no cover + logSys.log(5, " Looking for prefregex %r", self.__prefRegex.getRegex()) + self.__prefRegex.search(self.__lineBuffer) + if not self.__prefRegex.hasMatched(): + logSys.log(5, " Prefregex not matched") + return failList + preGroups = self.__prefRegex.getGroups() + logSys.log(7, " Pre-filter matched %s", preGroups) + repl = preGroups.get('content') + # Content replacement: + if repl: + del preGroups['content'] + self.__lineBuffer = [('', '', repl)] # Iterates over all the regular expressions. for failRegexIndex, failRegex in enumerate(self.__failRegex): - failRegex.search(self.__lineBuffer) - if failRegex.hasMatched(): - # The failregex matched. - logSys.log(7, "Matched %s", failRegex) - # Checks if we must ignore this match. - if self.ignoreLine(failRegex.getMatchedTupleLines()) \ - is not None: - # The ignoreregex matched. Remove ignored match. - self.__lineBuffer = failRegex.getUnmatchedTupleLines() - logSys.log(7, "Matched ignoreregex and was ignored") - if not self.checkAllRegex: - break - else: - continue - if date is None: - logSys.warning( - "Found a match for %r but no valid date/time " - "found for %r. Please try setting a custom " - "date pattern (see man page jail.conf(5)). " - "If format is complex, please " - "file a detailed issue on" - " https://github.com/fail2ban/fail2ban/issues " - "in order to get support for this format." - % ("\n".join(failRegex.getMatchedLines()), timeText)) + if logSys.getEffectiveLevel() <= logging.HEAVYDEBUG: # pragma: no cover + logSys.log(5, " Looking for failregex %r", failRegex.getRegex()) + failRegex.search(self.__lineBuffer, orgBuffer) + if not failRegex.hasMatched(): + continue + # The failregex matched. + logSys.log(7, " Matched %s", failRegex) + # Checks if we must ignore this match. + if self.ignoreLine(failRegex.getMatchedTupleLines()) \ + is not None: + # The ignoreregex matched. Remove ignored match. + self.__lineBuffer = failRegex.getUnmatchedTupleLines() + logSys.log(7, " Matched ignoreregex and was ignored") + if not self.checkAllRegex: + break else: - self.__lineBuffer = failRegex.getUnmatchedTupleLines() - # retrieve failure-id, host, etc from failure match: - raw = returnRawHost - try: - fail = failRegex.getGroups() - # failure-id: - fid = fail.get('fid') - # ip-address or host: - host = fail.get('ip4') or fail.get('ip6') - if host is not None: - raw = True - else: - host = fail.get('dns') - if host is None: + continue + if date is None: + logSys.warning( + "Found a match for %r but no valid date/time " + "found for %r. Please try setting a custom " + "date pattern (see man page jail.conf(5)). " + "If format is complex, please " + "file a detailed issue on" + " https://github.com/fail2ban/fail2ban/issues " + "in order to get support for this format.", + "\n".join(failRegex.getMatchedLines()), timeText) + continue + self.__lineBuffer = failRegex.getUnmatchedTupleLines() + # retrieve failure-id, host, etc from failure match: + try: + raw = returnRawHost + if preGroups: + fail = preGroups.copy() + fail.update(failRegex.getGroups()) + else: + fail = failRegex.getGroups() + # first try to check we have mlfid case (caching of connection id by multi-line): + mlfid = fail.get('mlfid') + if mlfid is not None: + fail = self._mergeFailure(mlfid, fail, failRegex) + # bypass if no-failure case: + if fail.get('nofail'): + logSys.log(7, "Nofail by mlfid %r in regex %s: %s", + mlfid, failRegexIndex, fail.get('mlfforget', "waiting for failure")) + if not self.checkAllRegex: return failList + else: + # matched lines: + fail["matches"] = fail.get("matches", []) + failRegex.getMatchedTupleLines() + # failure-id: + fid = fail.get('fid') + # ip-address or host: + host = fail.get('ip4') + if host is not None: + cidr = IPAddr.FAM_IPv4 + raw = True + else: + host = fail.get('ip6') + if host is not None: + cidr = IPAddr.FAM_IPv6 + raw = True + if host is None: + host = fail.get('dns') + if host is None: + # first try to check we have mlfid case (cache connection id): + if fid is None and mlfid is None: # if no failure-id also (obscure case, wrong regex), throw error inside getFailID: - if fid is None: - fid = failRegex.getFailID() - host = fid - cidr = IPAddr.CIDR_RAW - # if raw - add single ip or failure-id, - # otherwise expand host to multiple ips using dns (or ignore it if not valid): - if raw: - ip = IPAddr(host, cidr) - # check host equal failure-id, if not - failure with complex id: - if fid is not None and fid != host: - ip = IPAddr(fid, IPAddr.CIDR_RAW) - failList.append([failRegexIndex, ip, date, - failRegex.getMatchedLines(), fail]) - if not self.checkAllRegex: - break - else: - ips = DNSUtils.textToIp(host, self.__useDns) - if ips: - for ip in ips: - failList.append([failRegexIndex, ip, date, - failRegex.getMatchedLines(), fail]) - if not self.checkAllRegex: - break - except RegexException as e: # pragma: no cover - unsure if reachable - logSys.error(e) + fid = failRegex.getFailID() + host = fid + cidr = IPAddr.CIDR_RAW + # if mlfid case (not failure): + if host is None: + logSys.log(7, "No failure-id by mlfid %r in regex %s: %s", + mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier")) + if not self.checkAllRegex: return failList + ips = [None] + # if raw - add single ip or failure-id, + # otherwise expand host to multiple ips using dns (or ignore it if not valid): + elif raw: + ip = IPAddr(host, cidr) + # check host equal failure-id, if not - failure with complex id: + if fid is not None and fid != host: + ip = IPAddr(fid, IPAddr.CIDR_RAW) + ips = [ip] + # otherwise, try to use dns conversion: + else: + ips = DNSUtils.textToIp(host, self.__useDns) + # append failure with match to the list: + for ip in ips: + failList.append([failRegexIndex, ip, date, fail]) + if not self.checkAllRegex: + break + except RegexException as e: # pragma: no cover - unsure if reachable + logSys.error(e) return failList def status(self, flavor="basic"): @@ -716,7 +846,7 @@ class FileFilter(Filter): db = self.jail.database if db is not None: db.updateLog(self.jail, log) - logSys.info("Removed logfile: %r" % path) + logSys.info("Removed logfile: %r", path) self._delLogPath(path) return @@ -781,7 +911,7 @@ class FileFilter(Filter): def getFailures(self, filename): log = self.getLog(filename) if log is None: - logSys.error("Unable to get failures in " + filename) + logSys.error("Unable to get failures in %s", filename) return False # We should always close log (file), otherwise may be locked (log-rotate, etc.) try: @@ -790,11 +920,12 @@ class FileFilter(Filter): has_content = log.open() # see http://python.org/dev/peps/pep-3151/ except IOError as e: - logSys.error("Unable to open %s" % filename) - logSys.exception(e) + logSys.error("Unable to open %s", filename) + if e.errno != 2: # errno.ENOENT + logSys.exception(e) return False except OSError as e: # pragma: no cover - requires race condition to tigger this - logSys.error("Error opening %s" % filename) + logSys.error("Error opening %s", filename) logSys.exception(e) return False except Exception as e: # pragma: no cover - Requires implemention error in FileContainer to generate @@ -867,7 +998,9 @@ class FileFilter(Filter): break (timeMatch, template) = self.dateDetector.matchTime(line) if timeMatch: - dateTimeMatch = self.dateDetector.getTime(line[timeMatch.start():timeMatch.end()], (timeMatch, template)) + dateTimeMatch = self.dateDetector.getTime( + line[timeMatch.start():timeMatch.end()], + (timeMatch, template)) else: nextp = container.tell() if nextp > maxp: @@ -1015,7 +1148,7 @@ class FileContainer: ## sys.stdout.flush() # Compare hash and inode if self.__hash != myHash or self.__ino != stats.st_ino: - logSys.info("Log rotation detected for %s" % self.__filename) + logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename) self.__hash = myHash self.__ino = stats.st_ino self.__pos = 0 diff --git a/fail2ban/server/filtergamin.py b/fail2ban/server/filtergamin.py index 106e4c0f..3baf8c54 100644 --- a/fail2ban/server/filtergamin.py +++ b/fail2ban/server/filtergamin.py @@ -143,6 +143,8 @@ class FilterGamin(FileFilter): # Desallocates the resources used by Gamin. def __cleanup(self): + if not self.monitor: + return for filename in self.getLogPaths(): self.monitor.stop_watch(filename) self.monitor = None diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py index 73c82099..41b8be7c 100644 --- a/fail2ban/server/filterpyinotify.py +++ b/fail2ban/server/filterpyinotify.py @@ -25,19 +25,20 @@ __license__ = "GPL" import logging from distutils.version import LooseVersion +import os from os.path import dirname, sep as pathsep import pyinotify from .failmanager import FailManagerEmpty from .filter import FileFilter -from .mytime import MyTime +from .mytime import MyTime, time from .utils import Utils from ..helpers import getLogger if not hasattr(pyinotify, '__version__') \ - or LooseVersion(pyinotify.__version__) < '0.8.3': + or LooseVersion(pyinotify.__version__) < '0.8.3': # pragma: no cover raise ImportError("Fail2Ban requires pyinotify >= 0.8.3") # Verify that pyinotify is functional on this system @@ -45,13 +46,18 @@ if not hasattr(pyinotify, '__version__') \ try: manager = pyinotify.WatchManager() del manager -except Exception as e: +except Exception as e: # pragma: no cover raise ImportError("Pyinotify is probably not functional on this system: %s" % str(e)) # Gets the instance of the logger. logSys = getLogger(__name__) +# Override pyinotify default logger/init-handler: +def _pyinotify_logger_init(): # pragma: no cover + return logSys +pyinotify._logger_init = _pyinotify_logger_init +pyinotify.log = logSys ## # Log reader class. @@ -72,30 +78,57 @@ class FilterPyinotify(FileFilter): self.__modified = False # Pyinotify watch manager self.__monitor = pyinotify.WatchManager() - self.__watches = dict() + self.__watchFiles = dict() + self.__watchDirs = dict() + self.__pending = dict() + self.__pendingChkTime = 0 + self.__pendingMinTime = 60 logSys.debug("Created FilterPyinotify") def callback(self, event, origin=''): logSys.log(7, "[%s] %sCallback for Event: %s", self.jailName, origin, event) path = event.pathname + # check watching of this path: + isWF = False + isWD = path in self.__watchDirs + if not isWD and path in self.__watchFiles: + isWF = True + assumeNoDir = False if event.mask & ( pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO ): # skip directories altogether if event.mask & pyinotify.IN_ISDIR: logSys.debug("Ignoring creation of directory %s", path) return # check if that is a file we care about - if not path in self.__watches: + if not isWF: logSys.debug("Ignoring creation of %s we do not monitor", path) return - else: - # we need to substitute the watcher with a new one, so first - # remove old one - self._delFileWatcher(path) - # place a new one - self._addFileWatcher(path) + self._refreshWatcher(path) + elif event.mask & (pyinotify.IN_IGNORED | pyinotify.IN_MOVE_SELF | pyinotify.IN_DELETE_SELF): + assumeNoDir = event.mask & (pyinotify.IN_MOVE_SELF | pyinotify.IN_DELETE_SELF) + # fix pyinotify behavior with '-unknown-path' (if target not watched also): + if (assumeNoDir and + path.endswith('-unknown-path') and not isWF and not isWD + ): + path = path[:-len('-unknown-path')] + isWD = path in self.__watchDirs + # watch was removed for some reasons (log-rotate?): + if isWD and (assumeNoDir or not os.path.isdir(path)): + self._addPending(path, event, isDir=True) + elif not isWF: # pragma: no cover (assume too sporadic) + for logpath in self.__watchDirs: + if logpath.startswith(path + pathsep) and (assumeNoDir or not os.path.isdir(logpath)): + self._addPending(logpath, event, isDir=True) + if isWF and not os.path.isfile(path): + self._addPending(path, event) + return # do nothing if idle: if self.idle: return + # be sure we process a file: + if not isWF: + logSys.debug("Ignoring event (%s) of %s we do not monitor", event.maskname, path) + return self._process_file(path) def _process_file(self, path): @@ -104,23 +137,94 @@ class FilterPyinotify(FileFilter): TODO -- RF: this is a common logic and must be shared/provided by FileFilter """ - self.getFailures(path) + if not self.idle: + self.getFailures(path) + try: + while True: + ticket = self.failManager.toBan() + self.jail.putFailTicket(ticket) + except FailManagerEmpty: + self.failManager.cleanup(MyTime.time()) + self.__modified = False + + def _addPending(self, path, reason, isDir=False): + if path not in self.__pending: + self.__pending[path] = [Utils.DEFAULT_SLEEP_INTERVAL, isDir]; + self.__pendingMinTime = 0 + if isinstance(reason, pyinotify.Event): + reason = [reason.maskname, reason.pathname] + logSys.log(logging.MSG, "Log absence detected (possibly rotation) for %s, reason: %s of %s", + path, *reason) + + def _delPending(self, path): try: - while True: - ticket = self.failManager.toBan() - self.jail.putFailTicket(ticket) - except FailManagerEmpty: - self.failManager.cleanup(MyTime.time()) - self.__modified = False + del self.__pending[path] + except KeyError: pass + + def _checkPending(self): + if not self.__pending: + return + ntm = time.time() + if ntm < self.__pendingChkTime + self.__pendingMinTime: + return + found = {} + minTime = 60 + for path, (retardTM, isDir) in self.__pending.iteritems(): + if ntm - self.__pendingChkTime < retardTM: + if minTime > retardTM: minTime = retardTM + continue + chkpath = os.path.isdir if isDir else os.path.isfile + if not chkpath(path): # not found - prolong for next time + if retardTM < 60: retardTM *= 2 + if minTime > retardTM: minTime = retardTM + self.__pending[path][0] = retardTM + continue + logSys.log(logging.MSG, "Log presence detected for %s %s", + "directory" if isDir else "file", path) + found[path] = isDir + self.__pendingChkTime = time.time() + self.__pendingMinTime = minTime + # process now because we've missed it in monitoring: + for path, isDir in found.iteritems(): + self._delPending(path) + # refresh monitoring of this: + self._refreshWatcher(path, isDir=isDir) + if isDir: + # check all files belong to this dir: + for logpath in self.__watchFiles: + if logpath.startswith(path + pathsep): + # if still no file - add to pending, otherwise refresh and process: + if not os.path.isfile(logpath): + self._addPending(logpath, ('FROM_PARDIR', path)) + else: + self._refreshWatcher(logpath) + self._process_file(logpath) + else: + # process (possibly no old events for it from watcher): + self._process_file(path) + + def _refreshWatcher(self, oldPath, newPath=None, isDir=False): + if not newPath: newPath = oldPath + # we need to substitute the watcher with a new one, so first + # remove old one and then place a new one + if not isDir: + self._delFileWatcher(oldPath) + self._addFileWatcher(newPath) + else: + self._delDirWatcher(oldPath) + self._addDirWatcher(newPath) def _addFileWatcher(self, path): + # we need to watch also the directory for IN_CREATE + self._addDirWatcher(dirname(path)) + # add file watcher: wd = self.__monitor.add_watch(path, pyinotify.IN_MODIFY) - self.__watches.update(wd) + self.__watchFiles.update(wd) logSys.debug("Added file watcher for %s", path) def _delFileWatcher(self, path): try: - wdInt = self.__watches.pop(path) + wdInt = self.__watchFiles.pop(path) wd = self.__monitor.rm_watch(wdInt) if wd[wdInt]: logSys.debug("Removed file watcher for %s", path) @@ -129,19 +233,30 @@ class FilterPyinotify(FileFilter): pass return False + def _addDirWatcher(self, path_dir): + # Add watch for the directory: + if path_dir not in self.__watchDirs: + self.__watchDirs.update( + self.__monitor.add_watch(path_dir, pyinotify.IN_CREATE | + pyinotify.IN_MOVED_TO | pyinotify.IN_MOVE_SELF | + pyinotify.IN_DELETE_SELF | pyinotify.IN_ISDIR)) + logSys.debug("Added monitor for the parent directory %s", path_dir) + + def _delDirWatcher(self, path_dir): + # Remove watches for the directory: + try: + wdInt = self.__watchDirs.pop(path_dir) + self.__monitor.rm_watch(wdInt) + except KeyError: # pragma: no cover + pass + logSys.debug("Removed monitor for the parent directory %s", path_dir) + ## # Add a log file path # # @param path log file path def _addLogPath(self, path): - path_dir = dirname(path) - if not (path_dir in self.__watches): - # we need to watch also the directory for IN_CREATE - self.__watches.update( - self.__monitor.add_watch(path_dir, pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO)) - logSys.debug("Added monitor for the parent directory %s", path_dir) - self._addFileWatcher(path) self._process_file(path) @@ -151,39 +266,37 @@ class FilterPyinotify(FileFilter): # @param path the log file to delete def _delLogPath(self, path): - if not self._delFileWatcher(path): + if not self._delFileWatcher(path): # pragma: no cover logSys.error("Failed to remove watch on path: %s", path) + self._delPending(path) path_dir = dirname(path) - if not len([k for k in self.__watches - if k.startswith(path_dir + pathsep)]): + for k in self.__watchFiles: + if k.startswith(path_dir + pathsep): + path_dir = None + break + if path_dir: # Remove watches for the directory # since there is no other monitored file under this directory - try: - wdInt = self.__watches.pop(path_dir) - self.__monitor.rm_watch(wdInt) - except KeyError: # pragma: no cover - pass - logSys.debug("Removed monitor for the parent directory %s", path_dir) + self._delDirWatcher(path_dir) + self._delPending(path_dir) # pyinotify.ProcessEvent default handler: def __process_default(self, event): try: self.callback(event, origin='Default ') - except Exception as e: + except Exception as e: # pragma: no cover logSys.error("Error in FilterPyinotify callback: %s", e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG) + # incr common error counter: + self.commonError() self.ticks += 1 - # slow check events while idle: - def __check_events(self, *args, **kwargs): - if self.idle: - if Utils.wait_for(lambda: not self.active or not self.idle, - self.sleeptime * 10, self.sleeptime - ): - pass - self.ticks += 1 - return pyinotify.ThreadedNotifier.check_events(self.__notifier, *args, **kwargs) + @property + def __notify_maxtout(self): + # timeout for pyinotify must be set in milliseconds (fail2ban time values are + # floats contain seconds), max 0.5 sec (additionally regards pending check time) + return min(self.sleeptime, 0.5, self.__pendingMinTime) * 1000 ## # Main loop. @@ -194,26 +307,64 @@ class FilterPyinotify(FileFilter): def run(self): prcevent = pyinotify.ProcessEvent() prcevent.process_default = self.__process_default - ## timeout for pyinotify must be set in milliseconds (our time values are floats contain seconds) - self.__notifier = pyinotify.ThreadedNotifier(self.__monitor, - prcevent, timeout=self.sleeptime * 1000) - self.__notifier.check_events = self.__check_events - self.__notifier.start() + self.__notifier = pyinotify.Notifier(self.__monitor, + prcevent, timeout=self.__notify_maxtout) logSys.debug("[%s] filter started (pyinotifier)", self.jailName) + while self.active: + try: + + # slow check events while idle: + if self.idle: + if Utils.wait_for(lambda: not self.active or not self.idle, + min(self.sleeptime * 10, self.__pendingMinTime), + min(self.sleeptime, self.__pendingMinTime) + ): + if not self.active: break + + # default pyinotify handling using Notifier: + self.__notifier.process_events() + + # wait for events / timeout: + notify_maxtout = self.__notify_maxtout + def __check_events(): + return not self.active or self.__notifier.check_events(timeout=notify_maxtout) + if Utils.wait_for(__check_events, min(self.sleeptime, self.__pendingMinTime)): + if not self.active: break + self.__notifier.read_events() + + # check pending files/dirs (logrotate ready): + if not self.idle: + self._checkPending() + + except Exception as e: # pragma: no cover + if not self.active: # if not active - error by stop... + break + logSys.error("Caught unhandled exception in main cycle: %r", e, + exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) + # incr common error counter: + self.commonError() + + self.ticks += 1 + + logSys.debug("[%s] filter exited (pyinotifier)", self.jailName) + self.__notifier = None + return True ## # Call super.stop() and then stop the 'Notifier' def stop(self): + # stop filter thread: super(FilterPyinotify, self).stop() - # Stop the notifier thread - self.__notifier.stop() + if self.__notifier: # stop the notifier + self.__notifier.stop() ## # Wait for exit with cleanup. def join(self): + self.join = lambda *args: 0 self.__cleanup() super(FilterPyinotify, self).join() logSys.debug("[%s] filter terminated (pyinotifier)", self.jailName) @@ -223,6 +374,6 @@ class FilterPyinotify(FileFilter): def __cleanup(self): if self.__notifier: - self.__notifier.join() # to not exit before notifier does - self.__notifier = None - self.__monitor = None + if Utils.wait_for(lambda: not self.__notifier, self.sleeptime * 10): + self.__notifier = None + self.__monitor = None diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 0d720a5e..f83e7264 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -179,6 +179,14 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover return self.__matches ## + # Get journal reader + # + # @return journal reader + + def getJournalReader(self): + return self.__journal + + ## # Format journal log entry into syslog style # # @param entry systemd journal entry dict @@ -307,6 +315,7 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover self.commonError() logSys.debug("[%s] filter terminated", self.jailName) + # close journal: try: if self.__journal: @@ -314,8 +323,8 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover except Exception as e: # pragma: no cover logSys.error("Close journal failed: %r", e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) - logSys.debug((self.jail is not None and self.jail.name - or "jailless") +" filter terminated") + + logSys.debug("[%s] filter exited (systemd)", self.jailName) return True def status(self, flavor="basic"): diff --git a/fail2ban/server/ipdns.py b/fail2ban/server/ipdns.py index f8db6a04..6ef36888 100644 --- a/fail2ban/server/ipdns.py +++ b/fail2ban/server/ipdns.py @@ -64,16 +64,19 @@ class DNSUtils: if ips is not None: return ips # retrieve ips - try: - ips = list() - for result in socket.getaddrinfo(dns, None, 0, 0, socket.IPPROTO_TCP): - ip = IPAddr(result[4][0]) - if ip.isValid: - ips.append(ip) - except socket.error as e: - # todo: make configurable the expired time of cache entry: - logSys.warning("Unable to find a corresponding IP address for %s: %s", dns, e) - ips = list() + ips = list() + saveerr = None + for fam, ipfam in ((socket.AF_INET, IPAddr.FAM_IPv4), (socket.AF_INET6, IPAddr.FAM_IPv6)): + try: + for result in socket.getaddrinfo(dns, None, fam, 0, socket.IPPROTO_TCP): + ip = IPAddr(result[4][0], ipfam) + if ip.isValid: + ips.append(ip) + except socket.error as e: + saveerr = e + if not ips and saveerr: + logSys.warning("Unable to find a corresponding IP address for %s: %s", dns, saveerr) + DNSUtils.CACHE_nameToIp.set(dns, ips) return ips @@ -115,6 +118,60 @@ class DNSUtils: return ipList + @staticmethod + def getHostname(fqdn=True): + """Get short hostname or fully-qualified hostname of host self""" + # try find cached own hostnames (this tuple-key cannot be used elsewhere): + key = ('self','hostname', fqdn) + name = DNSUtils.CACHE_ipToName.get(key) + # get it using different ways (hostname, fully-qualified or vice versa): + if name is None: + name = '' + for hostname in ( + (socket.getfqdn, socket.gethostname) if fqdn else (socket.gethostname, socket.getfqdn) + ): + try: + name = hostname() + break + except Exception as e: # pragma: no cover + logSys.warning("Retrieving own hostnames failed: %s", e) + # cache and return : + DNSUtils.CACHE_ipToName.set(key, name) + return name + + @staticmethod + def getSelfNames(): + """Get own host names of self""" + # try find cached own hostnames (this tuple-key cannot be used elsewhere): + key = ('self','dns') + names = DNSUtils.CACHE_ipToName.get(key) + # get it using different ways (a set with names of localhost, hostname, fully qualified): + if names is None: + names = set([ + 'localhost', DNSUtils.getHostname(False), DNSUtils.getHostname(True) + ]) - set(['']) # getHostname can return '' + # cache and return : + DNSUtils.CACHE_ipToName.set(key, names) + return names + + @staticmethod + def getSelfIPs(): + """Get own IP addresses of self""" + # try find cached own IPs (this tuple-key cannot be used elsewhere): + key = ('self','ips') + ips = DNSUtils.CACHE_nameToIp.get(key) + # get it using different ways (a set with IPs of localhost, hostname, fully qualified): + if ips is None: + ips = set() + for hostname in DNSUtils.getSelfNames(): + try: + ips |= set(DNSUtils.textToIp(hostname, 'yes')) + except Exception as e: # pragma: no cover + logSys.warning("Retrieving own IPs of %s failed: %s", hostname, e) + # cache and return : + DNSUtils.CACHE_nameToIp.set(key, ips) + return ips + ## # Class for IP address handling. @@ -140,6 +197,8 @@ class IPAddr(object): CIDR_RAW = -2 CIDR_UNSPEC = -1 + FAM_IPv4 = CIDR_RAW - socket.AF_INET + FAM_IPv6 = CIDR_RAW - socket.AF_INET6 def __new__(cls, ipstr, cidr=CIDR_UNSPEC): # check already cached as IPAddr @@ -191,7 +250,11 @@ class IPAddr(object): self._raw = ipstr # if not raw - recognize family, set addr, etc.: if cidr != IPAddr.CIDR_RAW: - for family in [socket.AF_INET, socket.AF_INET6]: + if cidr is not None and cidr < IPAddr.CIDR_RAW: + family = [IPAddr.CIDR_RAW - cidr] + else: + family = [socket.AF_INET, socket.AF_INET6] + for family in family: try: binary = socket.inet_pton(family, ipstr) self._family = family @@ -252,6 +315,11 @@ class IPAddr(object): def family(self): return self._family + FAM2STR = {socket.AF_INET: 'inet4', socket.AF_INET6: 'inet6'} + @property + def familyStr(self): + return IPAddr.FAM2STR.get(self._family) + @property def plen(self): return self._plen @@ -346,7 +414,7 @@ class IPAddr(object): return socket.inet_ntop(self._family, binary) + add - def getPTR(self, suffix=""): + def getPTR(self, suffix=None): """ return the DNS PTR string of the provided IP address object If "suffix" is provided it will be appended as the second and top @@ -356,17 +424,22 @@ class IPAddr(object): """ if self.isIPv4: exploded_ip = self.ntoa.split(".") - if not suffix: + if suffix is None: suffix = "in-addr.arpa." elif self.isIPv6: exploded_ip = self.hexdump - if not suffix: + if suffix is None: suffix = "ip6.arpa." else: return "" return "%s.%s" % (".".join(reversed(exploded_ip)), suffix) + def getHost(self): + """Return the host name (DNS) of the provided IP address object + """ + return DNSUtils.ipToName(self.ntoa) + @property def isIPv4(self): """Either the IP object is of address family AF_INET diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index 313b6ee5..e3b22c44 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -308,6 +308,12 @@ class Server: return self.__jails[name].idle # Filter + def setIgnoreSelf(self, name, value): + self.__jails[name].filter.ignoreSelf = value + + def getIgnoreSelf(self, name): + return self.__jails[name].filter.ignoreSelf + def addIgnoreIP(self, name, ip): self.__jails[name].filter.addIgnoreIP(ip) @@ -373,12 +379,26 @@ class Server: def getDatePattern(self, name): return self.__jails[name].filter.getDatePattern() + def setLogTimeZone(self, name, tz): + self.__jails[name].filter.setLogTimeZone(tz) + + def getLogTimeZone(self, name): + return self.__jails[name].filter.getLogTimeZone() + def setIgnoreCommand(self, name, value): self.__jails[name].filter.setIgnoreCommand(value) def getIgnoreCommand(self, name): return self.__jails[name].filter.getIgnoreCommand() + def setPrefRegex(self, name, value): + flt = self.__jails[name].filter + logSys.debug(" prefregex: %r", value) + flt.prefRegex = value + + def getPrefRegex(self, name): + return self.__jails[name].filter.prefRegex + def addFailRegex(self, name, value, multiple=False): flt = self.__jails[name].filter if not multiple: value = (value,) @@ -624,9 +644,9 @@ class Server: if self.__syslogSocket == syslogsocket: return True self.__syslogSocket = syslogsocket - # Conditionally reload, logtarget depends on socket path when SYSLOG - return self.__logTarget != "SYSLOG"\ - or self.setLogTarget(self.__logTarget) + # Conditionally reload, logtarget depends on socket path when SYSLOG + return self.__logTarget != "SYSLOG"\ + or self.setLogTarget(self.__logTarget) def getLogTarget(self): with self.__loggingLock: diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index cdfe0e0e..dbf75d21 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -17,6 +17,7 @@ # along with Fail2Ban; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +import re import time import calendar import datetime @@ -25,7 +26,9 @@ from _strptime import LocaleTime, TimeRE, _calc_julian_from_U_or_W from .mytime import MyTime locale_time = LocaleTime() -timeRE = TimeRE() + +TZ_ABBR_RE = r"[A-Z](?:[A-Z]{2,4})?" +FIXED_OFFSET_TZ_RE = re.compile(r"(%s)?([+-][01]\d(?::?\d{2})?)?$" % (TZ_ABBR_RE,)) def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNow)): """ Build century regex for last year and the next years (distance). @@ -38,10 +41,20 @@ def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNo exprset |= set( cent(now[1].year + i) for i in (-1, distance) ) return "(?:%s)" % "|".join(exprset) if len(exprset) > 1 else "".join(exprset) -#todo: implement literal time zone support like CET, PST, PDT, etc (via pytz): -#timeRE['z'] = r"%s?(?P<z>Z|[+-]\d{2}(?::?[0-5]\d)?|[A-Z]{3})?" % timeRE['Z'] -timeRE['Z'] = r"(?P<Z>[A-Z]{3,5})" -timeRE['z'] = r"(?P<z>Z|UTC|GMT|[+-]\d{2}(?::?[0-5]\d)?)" +timeRE = TimeRE() + +# TODO: because python currently does not support mixing of case-sensitive with case-insensitive matching, +# check how TZ (in uppercase) can be combined with %a/%b etc. (that are currently case-insensitive), +# to avoid invalid date-time recognition in strings like '11-Aug-2013 03:36:11.372 error ...' +# with wrong TZ "error", which is at least not backwards compatible. +# Hence %z currently match literal Z|UTC|GMT only (and offset-based), and %Exz - all zone abbreviations. +timeRE['Z'] = r"(?P<Z>Z|[A-Z]{3,5})" +timeRE['z'] = r"(?P<z>Z|UTC|GMT|[+-][01]\d(?::?\d{2})?)" + +# Note: this extended tokens supported zone abbreviations, but it can parse 1 or 3-5 char(s) in lowercase, +# see todo above. Don't use them in default date-patterns (if not anchored, few precise resp. optional). +timeRE['ExZ'] = r"(?P<Z>%s)" % (TZ_ABBR_RE,) +timeRE['Exz'] = r"(?P<z>(?:%s)?[+-][01]\d(?::?\d{2})?|%s)" % (TZ_ABBR_RE, TZ_ABBR_RE) # Extend build-in TimeRE with some exact patterns # exact two-digit patterns: @@ -78,7 +91,56 @@ def getTimePatternRE(): names[key] = "%%%s" % key return (patt, names) -def reGroupDictStrptime(found_dict, msec=False): + +def validateTimeZone(tz): + """Validate a timezone and convert it to offset if it can (offset-based TZ). + + For now this accepts the UTC[+-]hhmm format (UTC has aliases GMT/Z and optional). + Additionally it accepts all zone abbreviations mentioned below in TZ_STR. + Note that currently this zone abbreviations are offset-based and used fixed + offset without automatically DST-switch (if CET used then no automatically CEST-switch). + + In the future, it may be extended for named time zones (such as Europe/Paris) + present on the system, if a suitable tz library is present (pytz). + """ + if tz is None: + return None + m = FIXED_OFFSET_TZ_RE.match(tz) + if m is None: + raise ValueError("Unknown or unsupported time zone: %r" % tz) + tz = m.groups() + return zone2offset(tz, 0) + +def zone2offset(tz, dt): + """Return the proper offset, in minutes according to given timezone at a given time. + + Parameters + ---------- + tz: symbolic timezone or offset (for now only TZA?([+-]hh:?mm?)? is supported, + as value are accepted: + int offset; + string in form like 'CET+0100' or 'UTC' or '-0400'; + tuple (or list) in form (zone name, zone offset); + dt: datetime instance for offset computation (currently unused) + """ + if isinstance(tz, int): + return tz + if isinstance(tz, basestring): + return validateTimeZone(tz) + tz, tzo = tz + if tzo is None or tzo == '': # without offset + return TZ_ABBR_OFFS[tz] + if len(tzo) <= 3: # short tzo (hh only) + # [+-]hh --> [+-]hh*60 + return TZ_ABBR_OFFS[tz] + int(tzo)*60 + if tzo[3] != ':': + # [+-]hhmm --> [+-]1 * (hh*60 + mm) + return TZ_ABBR_OFFS[tz] + (-1 if tzo[0] == '-' else 1) * (int(tzo[1:3])*60 + int(tzo[3:5])) + else: + # [+-]hh:mm --> [+-]1 * (hh*60 + mm) + return TZ_ABBR_OFFS[tz] + (-1 if tzo[0] == '-' else 1) * (int(tzo[1:3])*60 + int(tzo[4:6])) + +def reGroupDictStrptime(found_dict, msec=False, default_tz=None): """Return time from dictionary of strptime fields This is tweaked from python built-in _strptime. @@ -88,25 +150,18 @@ def reGroupDictStrptime(found_dict, msec=False): found_dict : dict Dictionary where keys represent the strptime fields, and values the respective value. - + default_tz : default timezone to apply if nothing relevant is in found_dict + (may be a non-fixed one in the future) Returns ------- float Unix time stamp. """ - now = MyTime.now() - year = month = day = hour = minute = None - hour = minute = None + now = \ + year = month = day = hour = minute = tzoffset = \ + weekday = julian = week_of_year = None second = fraction = 0 - tzoffset = None - # Default to -1 to signify that values not known; not critical to have, - # though - week_of_year = -1 - week_of_year_start = -1 - # weekday and julian defaulted to -1 so as to signal need to calculate - # values - weekday = julian = -1 for key, val in found_dict.iteritems(): if val is None: continue # Directives not explicitly handled below: @@ -116,13 +171,9 @@ def reGroupDictStrptime(found_dict, msec=False): # worthless without day of the week if key == 'y': year = int(val) - # Open Group specification for strptime() states that a %y - #value in the range of [00, 68] is in the century 2000, while - #[69,99] is in the century 1900 - if year <= 68: + # Fail2ban year should be always in the current century (>= 2000) + if year <= 2000: year += 2000 - else: - year += 1900 elif key == 'Y': year = int(val) elif key == 'm': @@ -156,7 +207,7 @@ def reGroupDictStrptime(found_dict, msec=False): elif key == 'S': second = int(val) elif key == 'f': - if msec: + if msec: # pragma: no cover - currently unused s = val # Pad to always return microseconds. s += "0" * (6 - len(s)) @@ -166,31 +217,20 @@ def reGroupDictStrptime(found_dict, msec=False): elif key == 'a': weekday = locale_time.a_weekday.index(val.lower()) elif key == 'w': - weekday = int(val) - if weekday == 0: - weekday = 6 - else: - weekday -= 1 + weekday = int(val) - 1 + if weekday < 0: weekday = 6 elif key == 'j': julian = int(val) elif key in ('U', 'W'): week_of_year = int(val) - if key == 'U': - # U starts week on Sunday. - week_of_year_start = 6 - else: - # W starts week on Monday. - week_of_year_start = 0 + # U starts week on Sunday, W - on Monday + week_of_year_start = 6 if key == 'U' else 0 elif key == 'z': z = val if z in ("Z", "UTC", "GMT"): tzoffset = 0 else: - tzoffset = int(z[1:3]) * 60 # Hours... - if len(z)>3: - tzoffset += int(z[-2:]) # ...and minutes - if z.startswith("-"): - tzoffset = -tzoffset + tzoffset = zone2offset(z, 0); # currently offset-based only elif key == 'Z': z = val if z in ("UTC", "GMT"): @@ -199,31 +239,28 @@ def reGroupDictStrptime(found_dict, msec=False): # Fail2Ban will assume it's this year assume_year = False if year is None: + if not now: now = MyTime.now() year = now.year assume_year = True - # If we know the week of the year and what day of that week, we can figure - # out the Julian day of the year. - if julian == -1 and week_of_year != -1 and weekday != -1: - week_starts_Mon = True if week_of_year_start == 0 else False - julian = _calc_julian_from_U_or_W(year, week_of_year, weekday, - week_starts_Mon) - # Cannot pre-calculate datetime.datetime() since can change in Julian - # calculation and thus could have different value for the day of the week - # calculation. - if julian != -1 and (month is None or day is None): - datetime_result = datetime.datetime.fromordinal((julian - 1) + datetime.datetime(year, 1, 1).toordinal()) - year = datetime_result.year - month = datetime_result.month - day = datetime_result.day - # Add timezone info - if tzoffset is not None: - gmtoff = tzoffset * 60 - else: - gmtoff = None + if month is None or day is None: + # If we know the week of the year and what day of that week, we can figure + # out the Julian day of the year. + if julian is None and week_of_year is not None and weekday is not None: + julian = _calc_julian_from_U_or_W(year, week_of_year, weekday, + (week_of_year_start == 0)) + # Cannot pre-calculate datetime.datetime() since can change in Julian + # calculation and thus could have different value for the day of the week + # calculation. + if julian is not None: + datetime_result = datetime.datetime.fromordinal((julian - 1) + datetime.datetime(year, 1, 1).toordinal()) + year = datetime_result.year + month = datetime_result.month + day = datetime_result.day # Fail2Ban assume today assume_today = False if month is None and day is None: + if not now: now = MyTime.now() month = now.month day = now.day assume_today = True @@ -231,22 +268,84 @@ def reGroupDictStrptime(found_dict, msec=False): # Actully create date date_result = datetime.datetime( year, month, day, hour, minute, second, fraction) - if gmtoff is not None: - date_result = date_result - datetime.timedelta(seconds=gmtoff) - - if date_result > now and assume_today: - # Rollover at midnight, could mean it's yesterday... - date_result = date_result - datetime.timedelta(days=1) - if date_result > now and assume_year: - # Could be last year? - # also reset month and day as it's not yesterday... - date_result = date_result.replace( - year=year-1, month=month, day=day) - - if gmtoff is not None: + # Correct timezone if not supplied in the log linge + if tzoffset is None and default_tz is not None: + tzoffset = zone2offset(default_tz, date_result) + # Add timezone info + if tzoffset is not None: + date_result -= datetime.timedelta(seconds=tzoffset * 60) + + if assume_today: + if not now: now = MyTime.now() + if date_result > now: + # Rollover at midnight, could mean it's yesterday... + date_result -= datetime.timedelta(days=1) + if assume_year: + if not now: now = MyTime.now() + if date_result > now: + # Could be last year? + # also reset month and day as it's not yesterday... + date_result = date_result.replace( + year=year-1, month=month, day=day) + + # make time: + if tzoffset is not None: tm = calendar.timegm(date_result.utctimetuple()) else: tm = time.mktime(date_result.timetuple()) - if msec: + if msec: # pragma: no cover - currently unused tm += fraction/1000000.0 return tm + + +TZ_ABBR_OFFS = {'':0, None:0} +TZ_STR = ''' + -12 Y + -11 X NUT SST + -10 W CKT HAST HST TAHT TKT + -9 V AKST GAMT GIT HADT HNY + -8 U AKDT CIST HAY HNP PST PT + -7 T HAP HNR MST PDT + -6 S CST EAST GALT HAR HNC MDT + -5 R CDT COT EASST ECT EST ET HAC HNE PET + -4 Q AST BOT CLT COST EDT FKT GYT HAE HNA PYT + -3 P ADT ART BRT CLST FKST GFT HAA PMST PYST SRT UYT WGT + -2 O BRST FNT PMDT UYST WGST + -1 N AZOT CVT EGT + 0 Z EGST GMT UTC WET WT + 1 A CET DFT WAT WEDT WEST + 2 B CAT CEDT CEST EET SAST WAST + 3 C EAT EEDT EEST IDT MSK + 4 D AMT AZT GET GST KUYT MSD MUT RET SAMT SCT + 5 E AMST AQTT AZST HMT MAWT MVT PKT TFT TJT TMT UZT YEKT + 6 F ALMT BIOT BTT IOT KGT NOVT OMST YEKST + 7 G CXT DAVT HOVT ICT KRAT NOVST OMSST THA WIB + 8 H ACT AWST BDT BNT CAST HKT IRKT KRAST MYT PHT SGT ULAT WITA WST + 9 I AWDT IRKST JST KST PWT TLT WDT WIT YAKT + 10 K AEST ChST PGT VLAT YAKST YAPT + 11 L AEDT LHDT MAGT NCT PONT SBT VLAST VUT + 12 M ANAST ANAT FJT GILT MAGST MHT NZST PETST PETT TVT WFT + 13 FJST NZDT + 11.5 NFT + 10.5 ACDT LHST + 9.5 ACST + 6.5 CCT MMT + 5.75 NPT + 5.5 SLT + 4.5 AFT IRDT + 3.5 IRST + -2.5 HAT NDT + -3.5 HNT NST NT + -4.5 HLV VET + -9.5 MART MIT +''' + +def _init_TZ_ABBR(): + """Initialized TZ_ABBR_OFFS dictionary (TZ -> offset in minutes)""" + for tzline in map(str.split, TZ_STR.split('\n')): + if not len(tzline): continue + tzoffset = int(float(tzline[0]) * 60) + for tz in tzline[1:]: + TZ_ABBR_OFFS[tz] = tzoffset + +_init_TZ_ABBR() diff --git a/fail2ban/server/ticket.py b/fail2ban/server/ticket.py index c7bb4d47..be205303 100644 --- a/fail2ban/server/ticket.py +++ b/fail2ban/server/ticket.py @@ -56,7 +56,9 @@ class Ticket(object): self._time = time if time is not None else MyTime.time() self._data = {'matches': matches or [], 'failures': 0} if data is not None: - self._data.update(data) + for k,v in data.iteritems(): + if v is not None: + self._data[k] = v if ticket: # ticket available - copy whole information from ticket: self.__dict__.update(i for i in ticket.__dict__.iteritems() if i[0] in self.__dict__) @@ -136,7 +138,8 @@ class Ticket(object): self._data['matches'] = matches or [] def getMatches(self): - return self._data.get('matches', []) + return [(line if isinstance(line, basestring) else "".join(line)) \ + for line in self._data.get('matches', ())] @property def restored(self): @@ -233,7 +236,11 @@ class FailTicket(Ticket): self.__retry += count self._data['failures'] += attempt if matches: - self._data['matches'] += matches + # we should duplicate "matches", because possibly referenced to multiple tickets: + if self._data['matches']: + self._data['matches'] = self._data['matches'] + matches + else: + self._data['matches'] = matches def setLastTime(self, value): if value > self._time: diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py index d23f12e2..ecc2a138 100644 --- a/fail2ban/server/transmitter.py +++ b/fail2ban/server/transmitter.py @@ -108,11 +108,11 @@ class Transmitter: value = command[1:] # if all ips: if len(value) == 1 and value[0] == "--all": - self.__server.setUnbanIP() - return + return self.__server.setUnbanIP() + cnt = 0 for value in value: - self.__server.setUnbanIP(None, value) - return None + cnt += self.__server.setUnbanIP(None, value) + return cnt elif command[0] == "echo": return command[1:] elif command[0] == "sleep": @@ -181,6 +181,10 @@ class Transmitter: raise Exception("Invalid idle option, must be 'on' or 'off'") return self.__server.getIdleJail(name) # Filter + elif command[1] == "ignoreself": + value = command[2] + self.__server.setIgnoreSelf(name, value) + return self.__server.getIgnoreSelf(name) elif command[1] == "addignoreip": value = command[2] self.__server.addIgnoreIP(name, value) @@ -221,6 +225,10 @@ class Transmitter: value = command[2:] self.__server.delJournalMatch(name, value) return self.__server.getJournalMatch(name) + elif command[1] == "prefregex": + value = command[2] + self.__server.setPrefRegex(name, value) + return self.__server.getPrefRegex(name) elif command[1] == "addfailregex": value = command[2] self.__server.addFailRegex(name, value, multiple=multiple) @@ -253,6 +261,10 @@ class Transmitter: value = command[2] self.__server.setDatePattern(name, value) return self.__server.getDatePattern(name) + elif command[1] == "logtimezone": + value = command[2] + self.__server.setLogTimeZone(name, value) + return self.__server.getLogTimeZone(name) elif command[1] == "maxretry": value = command[2] self.__server.setMaxRetry(name, int(value)) @@ -337,10 +349,14 @@ class Transmitter: return self.__server.getLogEncoding(name) elif command[1] == "journalmatch": # pragma: systemd no cover return self.__server.getJournalMatch(name) + elif command[1] == "ignoreself": + return self.__server.getIgnoreSelf(name) elif command[1] == "ignoreip": return self.__server.getIgnoreIP(name) elif command[1] == "ignorecommand": return self.__server.getIgnoreCommand(name) + elif command[1] == "prefregex": + return self.__server.getPrefRegex(name) elif command[1] == "failregex": return self.__server.getFailRegex(name) elif command[1] == "ignoreregex": @@ -351,6 +367,8 @@ class Transmitter: return self.__server.getFindTime(name) elif command[1] == "datepattern": return self.__server.getDatePattern(name) + elif command[1] == "logtimezone": + return self.__server.getLogTimeZone(name) elif command[1] == "maxretry": return self.__server.getMaxRetry(name) elif command[1] == "maxlines": diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py index 57da495a..58363ff0 100644 --- a/fail2ban/server/utils.py +++ b/fail2ban/server/utils.py @@ -28,7 +28,7 @@ import signal import subprocess import sys import time -from ..helpers import getLogger, uni_decode +from ..helpers import getLogger, _merge_dicts, uni_decode if sys.version_info >= (3, 3): import importlib.machinery @@ -60,6 +60,7 @@ class Utils(): DEFAULT_SLEEP_TIME = 2 DEFAULT_SLEEP_INTERVAL = 0.2 DEFAULT_SHORT_INTERVAL = 0.001 + DEFAULT_SHORTEST_INTERVAL = DEFAULT_SHORT_INTERVAL / 100 class Cache(object): @@ -98,6 +99,12 @@ class Utils(): cache.popitem() cache[k] = (v, t + self.maxTime) + def unset(self, k): + try: + del self._cache[k] + except KeyError: # pragme: no cover + pass + @staticmethod def setFBlockMode(fhandle, value): @@ -110,7 +117,31 @@ class Utils(): return flags @staticmethod - def executeCmd(realCmd, timeout=60, shell=True, output=False, tout_kill_tree=True, success_codes=(0,)): + def buildShellCmd(realCmd, varsDict): + """Generates new shell command as array, contains map as variables to + arguments statement (varsStat), the command (realCmd) used this variables and + the list of the arguments, mapped from varsDict + + Example: + buildShellCmd('echo "V2: $v2, V1: $v1"', {"v1": "val 1", "v2": "val 2", "vUnused": "unused var"}) + returns: + ['v1=$0 v2=$1 vUnused=$2 \necho "V2: $v2, V1: $v1"', 'val 1', 'val 2', 'unused var'] + """ + # build map as array of vars and command line array: + varsStat = "" + if not isinstance(realCmd, list): + realCmd = [realCmd] + i = len(realCmd)-1 + for k, v in varsDict.iteritems(): + varsStat += "%s=$%s " % (k, i) + realCmd.append(v) + i += 1 + realCmd[0] = varsStat + "\n" + realCmd[0] + return realCmd + + @staticmethod + def executeCmd(realCmd, timeout=60, shell=True, output=False, tout_kill_tree=True, + success_codes=(0,), varsDict=None): """Executes a command. Parameters @@ -125,6 +156,8 @@ class Utils(): output : bool If output is True, the function returns tuple (success, stdoutdata, stderrdata, returncode). If False, just indication of success is returned + varsDict: dict + variables supplied to the command (or to the shell script) Returns ------- @@ -140,10 +173,18 @@ class Utils(): """ stdout = stderr = None retcode = None - popen = None + popen = env = None + if varsDict: + if shell: + # build map as array of vars and command line array: + realCmd = Utils.buildShellCmd(realCmd, varsDict) + else: # pragma: no cover - currently unused + env = _merge_dicts(os.environ, varsDict) + realCmdId = id(realCmd) + logCmd = lambda level: logSys.log(level, "%x -- exec: %s", realCmdId, realCmd) try: popen = subprocess.Popen( - realCmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell, + realCmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell, env=env, preexec_fn=os.setsid # so that killpg does not kill our process ) # wait with timeout for process has terminated: @@ -152,13 +193,15 @@ class Utils(): def _popen_wait_end(): retcode = popen.poll() return (True, retcode) if retcode is not None else None - retcode = Utils.wait_for(_popen_wait_end, timeout, Utils.DEFAULT_SHORT_INTERVAL) + # popen.poll is fast operation so we can use the shortest sleep interval: + retcode = Utils.wait_for(_popen_wait_end, timeout, Utils.DEFAULT_SHORTEST_INTERVAL) if retcode: retcode = retcode[1] # if timeout: if retcode is None: - logSys.error("%s -- timed out after %s seconds." % - (realCmd, timeout)) + if logCmd: logCmd(logging.ERROR); logCmd = None + logSys.error("%x -- timed out after %s seconds." % + (realCmdId, timeout)) pgid = os.getpgid(popen.pid) # if not tree - first try to terminate and then kill, otherwise - kill (-9) only: os.killpg(pgid, signal.SIGTERM) # Terminate the process @@ -168,59 +211,62 @@ class Utils(): if retcode is None or tout_kill_tree: # Still going... os.killpg(pgid, signal.SIGKILL) # Kill the process time.sleep(Utils.DEFAULT_SLEEP_INTERVAL) - retcode = popen.poll() + if retcode is None: # pragma: no cover - too sporadic + retcode = popen.poll() #logSys.debug("%s -- killed %s ", realCmd, retcode) if retcode is None and not Utils.pid_exists(pgid): # pragma: no cover retcode = signal.SIGKILL except OSError as e: + if logCmd: logCmd(logging.ERROR); logCmd = None stderr = "%s -- failed with %s" % (realCmd, e) logSys.error(stderr) if not popen: return False if not output else (False, stdout, stderr, retcode) std_level = logging.DEBUG if retcode in success_codes else logging.ERROR + if std_level > logSys.getEffectiveLevel(): + if logCmd: logCmd(std_level-1); logCmd = None # if we need output (to return or to log it): if output or std_level >= logSys.getEffectiveLevel(): + # if was timeouted (killed/terminated) - to prevent waiting, set std handles to non-blocking mode. if popen.stdout: try: if retcode is None or retcode < 0: Utils.setFBlockMode(popen.stdout, False) stdout = popen.stdout.read() - except IOError as e: + except IOError as e: # pragma: no cover logSys.error(" ... -- failed to read stdout %s", e) if stdout is not None and stdout != '' and std_level >= logSys.getEffectiveLevel(): - logSys.log(std_level, "%s -- stdout:", realCmd) for l in stdout.splitlines(): - logSys.log(std_level, " -- stdout: %r", uni_decode(l)) + logSys.log(std_level, "%x -- stdout: %r", realCmdId, uni_decode(l)) popen.stdout.close() if popen.stderr: try: if retcode is None or retcode < 0: Utils.setFBlockMode(popen.stderr, False) stderr = popen.stderr.read() - except IOError as e: + except IOError as e: # pragma: no cover logSys.error(" ... -- failed to read stderr %s", e) if stderr is not None and stderr != '' and std_level >= logSys.getEffectiveLevel(): - logSys.log(std_level, "%s -- stderr:", realCmd) for l in stderr.splitlines(): - logSys.log(std_level, " -- stderr: %r", uni_decode(l)) + logSys.log(std_level, "%x -- stderr: %r", realCmdId, uni_decode(l)) popen.stderr.close() success = False if retcode in success_codes: - logSys.debug("%-.40s -- returned successfully %i", realCmd, retcode) + logSys.debug("%x -- returned successfully %i", realCmdId, retcode) success = True elif retcode is None: - logSys.error("%-.40s -- unable to kill PID %i", realCmd, popen.pid) + logSys.error("%x -- unable to kill PID %i", realCmdId, popen.pid) elif retcode < 0 or retcode > 128: # dash would return negative while bash 128 + n sigcode = -retcode if retcode < 0 else retcode - 128 - logSys.error("%-.40s -- killed with %s (return code: %s)", - realCmd, signame.get(sigcode, "signal %i" % sigcode), retcode) + logSys.error("%x -- killed with %s (return code: %s)", + realCmdId, signame.get(sigcode, "signal %i" % sigcode), retcode) else: msg = _RETCODE_HINTS.get(retcode, None) - logSys.error("%-.40s -- returned %i", realCmd, retcode) + logSys.error("%x -- returned %i", realCmdId, retcode) if msg: logSys.info("HINT on %i: %s", retcode, msg % locals()) if output: @@ -284,7 +330,7 @@ class Utils(): return e.errno == errno.EPERM else: return True - else: + else: # pragma : no cover (no windows currently supported) @staticmethod def pid_exists(pid): import ctypes diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py index 2f3b6723..2d08b5df 100644 --- a/fail2ban/tests/action_d/test_badips.py +++ b/fail2ban/tests/action_d/test_badips.py @@ -61,6 +61,7 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable # Must cancel timer! if self.action._timer: self.action._timer.cancel() + super(BadIPsActionTest, self).tearDown() def testCategory(self): categories = self.action.getCategories() diff --git a/fail2ban/tests/action_d/test_smtp.py b/fail2ban/tests/action_d/test_smtp.py index d9ad0f3a..d0858b85 100644 --- a/fail2ban/tests/action_d/test_smtp.py +++ b/fail2ban/tests/action_d/test_smtp.py @@ -30,18 +30,22 @@ else: from ..dummyjail import DummyJail -from ..utils import CONFIG_DIR, asyncserver - +from ..utils import CONFIG_DIR, asyncserver, Utils, uni_decode class TestSMTPServer(smtpd.SMTPServer): - def process_message(self, peer, mailfrom, rcpttos, data): + def __init__(self, *args): + smtpd.SMTPServer.__init__(self, *args) + self.ready = False + + def process_message(self, peer, mailfrom, rcpttos, data, **kwargs): self.peer = peer self.mailfrom = mailfrom self.rcpttos = rcpttos self.org_data = data - # replace new line (with tab or space) for possible mime translations (word wrap): - self.data = re.sub(r"\n[\t ]", " ", data) + # replace new line (with tab or space) for possible mime translations (word wrap), + self.data = re.sub(r"\n[\t ]", " ", uni_decode(data)) + self.ready = True class SMTPActionTest(unittest.TestCase): @@ -63,7 +67,7 @@ class SMTPActionTest(unittest.TestCase): port = self.smtpd.socket.getsockname()[1] self.action = customActionModule.Action( - self.jail, "test", host="127.0.0.1:%i" % port) + self.jail, "test", host="localhost:%i" % port) ## because of bug in loop (see loop in asyncserver.py) use it's loop instead of asyncore.loop: self._active = True @@ -77,9 +81,16 @@ class SMTPActionTest(unittest.TestCase): self.smtpd.close() self._active = False self._loop_thread.join() + super(SMTPActionTest, self).tearDown() + + def _exec_and_wait(self, doaction, timeout=3, short=False): + if short: timeout /= 25 + self.smtpd.ready = False + doaction() + Utils.wait_for(lambda: self.smtpd.ready, timeout) def testStart(self): - self.action.start() + self._exec_and_wait(self.action.start) self.assertEqual(self.smtpd.mailfrom, "fail2ban") self.assertEqual(self.smtpd.rcpttos, ["root"]) self.assertTrue( @@ -87,23 +98,28 @@ class SMTPActionTest(unittest.TestCase): in self.smtpd.data) def testStop(self): - self.action.stop() + self._exec_and_wait(self.action.stop) self.assertEqual(self.smtpd.mailfrom, "fail2ban") self.assertEqual(self.smtpd.rcpttos, ["root"]) self.assertTrue( "Subject: [Fail2Ban] %s: stopped" % self.jail.name in self.smtpd.data) - def testBan(self): + def _testBan(self, restored=False): aInfo = { 'ip': "127.0.0.2", 'failures': 3, 'matches': "Test fail 1\n", 'ipjailmatches': "Test fail 1\nTest Fail2\n", 'ipmatches': "Test fail 1\nTest Fail2\nTest Fail3\n", - } - - self.action.ban(aInfo) + } + if restored: + aInfo['restored'] = 1 + + self._exec_and_wait(lambda: self.action.ban(aInfo), short=restored) + if restored: # no mail, should raises attribute error: + self.assertRaises(AttributeError, lambda: self.smtpd.mailfrom) + return self.assertEqual(self.smtpd.mailfrom, "fail2ban") self.assertEqual(self.smtpd.rcpttos, ["root"]) subject = "Subject: [Fail2Ban] %s: banned %s" % ( @@ -113,26 +129,32 @@ class SMTPActionTest(unittest.TestCase): "%i attempts" % aInfo['failures'], self.smtpd.data) self.action.matches = "matches" - self.action.ban(aInfo) + self._exec_and_wait(lambda: self.action.ban(aInfo)) self.assertIn(aInfo['matches'], self.smtpd.data) self.action.matches = "ipjailmatches" - self.action.ban(aInfo) + self._exec_and_wait(lambda: self.action.ban(aInfo)) self.assertIn(aInfo['ipjailmatches'], self.smtpd.data) self.action.matches = "ipmatches" - self.action.ban(aInfo) + self._exec_and_wait(lambda: self.action.ban(aInfo)) self.assertIn(aInfo['ipmatches'], self.smtpd.data) + + def testBan(self): + self._testBan() + + def testNOPByRestored(self): + self._testBan(restored=True) def testOptions(self): - self.action.start() + self._exec_and_wait(self.action.start) self.assertEqual(self.smtpd.mailfrom, "fail2ban") self.assertEqual(self.smtpd.rcpttos, ["root"]) self.action.fromname = "Test" self.action.fromaddr = "test@example.com" self.action.toaddr = "test@example.com, test2@example.com" - self.action.start() + self._exec_and_wait(self.action.start) self.assertEqual(self.smtpd.mailfrom, "test@example.com") self.assertTrue("From: %s <%s>" % (self.action.fromname, self.action.fromaddr) in self.smtpd.data) diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py index 403f7ea6..cbd0aaca 100644 --- a/fail2ban/tests/actiontestcase.py +++ b/fail2ban/tests/actiontestcase.py @@ -29,7 +29,7 @@ import tempfile import time import unittest -from ..server.action import CommandAction, CallingMap +from ..server.action import CommandAction, CallingMap, substituteRecursiveTags from ..server.actions import OrderedDict from ..server.utils import Utils @@ -40,12 +40,20 @@ class CommandActionTest(LogCaptureTestCase): def setUp(self): """Call before every test case.""" - self.__action = CommandAction(None, "Test") LogCaptureTestCase.setUp(self) + self.__action = CommandAction(None, "Test") + # prevent execute stop if start fails (or event not started at all): + self.__action_started = False + orgstart = self.__action.start + def _action_start(): + self.__action_started = True + return orgstart() + self.__action.start = _action_start def tearDown(self): """Call after every test case.""" - self.__action.stop() + if self.__action_started: + self.__action.stop() LogCaptureTestCase.tearDown(self) def testSubstituteRecursiveTags(self): @@ -56,30 +64,30 @@ class CommandActionTest(LogCaptureTestCase): } # Recursion is bad self.assertRaises(ValueError, - lambda: CommandAction.substituteRecursiveTags({'A': '<A>'})) + lambda: substituteRecursiveTags({'A': '<A>'})) self.assertRaises(ValueError, - lambda: CommandAction.substituteRecursiveTags({'A': '<B>', 'B': '<A>'})) + lambda: substituteRecursiveTags({'A': '<B>', 'B': '<A>'})) self.assertRaises(ValueError, - lambda: CommandAction.substituteRecursiveTags({'A': '<B>', 'B': '<C>', 'C': '<A>'})) + lambda: substituteRecursiveTags({'A': '<B>', 'B': '<C>', 'C': '<A>'})) # Unresolveable substition self.assertRaises(ValueError, - lambda: CommandAction.substituteRecursiveTags({'A': 'to=<B> fromip=<IP>', 'C': '<B>', 'B': '<C>', 'D': ''})) + lambda: substituteRecursiveTags({'A': 'to=<B> fromip=<IP>', 'C': '<B>', 'B': '<C>', 'D': ''})) self.assertRaises(ValueError, - lambda: CommandAction.substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP>', 'sweet': '<honeypot>', 'honeypot': '<sweet>', 'ignoreregex': ''})) + lambda: substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP>', 'sweet': '<honeypot>', 'honeypot': '<sweet>', 'ignoreregex': ''})) # We need here an ordered, because the sequence of iteration is very important for this test if OrderedDict: # No cyclic recursion, just multiple replacement of tag <T>, should be successful: - self.assertEqual(CommandAction.substituteRecursiveTags( OrderedDict( + self.assertEqual(substituteRecursiveTags( OrderedDict( (('X', 'x=x<T>'), ('T', '1'), ('Z', '<X> <T> <Y>'), ('Y', 'y=y<T>'))) ), {'X': 'x=x1', 'T': '1', 'Y': 'y=y1', 'Z': 'x=x1 1 y=y1'} ) # No cyclic recursion, just multiple replacement of tag <T> in composite tags, should be successful: - self.assertEqual(CommandAction.substituteRecursiveTags( OrderedDict( + self.assertEqual(substituteRecursiveTags( OrderedDict( (('X', 'x=x<T> <Z> <<R1>> <<R2>>'), ('R1', 'Z'), ('R2', 'Y'), ('T', '1'), ('Z', '<T> <Y>'), ('Y', 'y=y<T>'))) ), {'X': 'x=x1 1 y=y1 1 y=y1 y=y1', 'R1': 'Z', 'R2': 'Y', 'T': '1', 'Z': '1 y=y1', 'Y': 'y=y1'} ) # No cyclic recursion, just multiple replacement of same tags, should be successful: - self.assertEqual(CommandAction.substituteRecursiveTags( OrderedDict(( + self.assertEqual(substituteRecursiveTags( OrderedDict(( ('actionstart', 'ipset create <ipmset> hash:ip timeout <bantime> family <ipsetfamily>\n<iptables> -I <chain> <actiontype>'), ('ipmset', 'f2b-<name>'), ('name', 'any'), @@ -111,44 +119,75 @@ class CommandActionTest(LogCaptureTestCase): )) ) # Cyclic recursion by composite tag creation, tags "create" another tag, that closes cycle: - self.assertRaises(ValueError, lambda: CommandAction.substituteRecursiveTags( OrderedDict(( + self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( ('A', '<<B><C>>'), ('B', 'D'), ('C', 'E'), ('DE', 'cycle <A>'), )) )) - self.assertRaises(ValueError, lambda: CommandAction.substituteRecursiveTags( OrderedDict(( + self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( ('DE', 'cycle <A>'), ('A', '<<B><C>>'), ('B', 'D'), ('C', 'E'), )) )) # missing tags are ok - self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C>'}), {'A': '<C>'}) - self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <D> <X>','X':'fun'}), {'A': '<C> <D> fun', 'X':'fun'}) - self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <B>', 'B': 'cool'}), {'A': '<C> cool', 'B': 'cool'}) + self.assertEqual(substituteRecursiveTags({'A': '<C>'}), {'A': '<C>'}) + self.assertEqual(substituteRecursiveTags({'A': '<C> <D> <X>','X':'fun'}), {'A': '<C> <D> fun', 'X':'fun'}) + self.assertEqual(substituteRecursiveTags({'A': '<C> <B>', 'B': 'cool'}), {'A': '<C> cool', 'B': 'cool'}) # Escaped tags should be ignored - self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<matches> <B>', 'B': 'cool'}), {'A': '<matches> cool', 'B': 'cool'}) + self.assertEqual(substituteRecursiveTags({'A': '<matches> <B>', 'B': 'cool'}), {'A': '<matches> cool', 'B': 'cool'}) # Multiple stuff on same line is ok - self.assertEqual(CommandAction.substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP> evilperson=<honeypot>', 'honeypot': 'pokie', 'ignoreregex': ''}), + self.assertEqual(substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP> evilperson=<honeypot>', 'honeypot': 'pokie', 'ignoreregex': ''}), { 'failregex': "to=pokie fromip=<IP> evilperson=pokie", 'honeypot': 'pokie', 'ignoreregex': '', }) # rest is just cool - self.assertEqual(CommandAction.substituteRecursiveTags(aInfo), + self.assertEqual(substituteRecursiveTags(aInfo), { 'HOST': "192.0.2.0", 'ABC': '123 192.0.2.0', 'xyz': '890 123 192.0.2.0', }) # obscure embedded case - self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<<PREF>HOST>', 'PREF': 'IPV4'}), + self.assertEqual(substituteRecursiveTags({'A': '<<PREF>HOST>', 'PREF': 'IPV4'}), {'A': '<IPV4HOST>', 'PREF': 'IPV4'}) - self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<<PREF>HOST>', 'PREF': 'IPV4', 'IPV4HOST': '1.2.3.4'}), + self.assertEqual(substituteRecursiveTags({'A': '<<PREF>HOST>', 'PREF': 'IPV4', 'IPV4HOST': '1.2.3.4'}), {'A': '1.2.3.4', 'PREF': 'IPV4', 'IPV4HOST': '1.2.3.4'}) # more embedded within a string and two interpolations - self.assertEqual(CommandAction.substituteRecursiveTags({'A': 'A <IP<PREF>HOST> B IP<PREF> C', 'PREF': 'V4', 'IPV4HOST': '1.2.3.4'}), + self.assertEqual(substituteRecursiveTags({'A': 'A <IP<PREF>HOST> B IP<PREF> C', 'PREF': 'V4', 'IPV4HOST': '1.2.3.4'}), {'A': 'A 1.2.3.4 B IPV4 C', 'PREF': 'V4', 'IPV4HOST': '1.2.3.4'}) + def testSubstRec_DontTouchUnusedCallable(self): + cm = CallingMap({ + 'A':0, + 'B':lambda self: '<A><A>', + 'C':'', + 'D':'' + }) + # + # should raise no exceptions: + substituteRecursiveTags(cm) + # add exception tag: + cm['C'] = lambda self,i=0: 5 // int(self['A']) # raise error by access + # test direct get of callable (should raise an error): + self.assertRaises(ZeroDivisionError, lambda: cm['C']) + # should raise no exceptions (tag "C" still unused): + substituteRecursiveTags(cm) + # add reference to "broken" tag: + cm['D'] = 'test=<C>' + # should raise an exception (BOOM by replacement of tag "D" recursive): + self.assertRaises(ZeroDivisionError, lambda: substituteRecursiveTags(cm)) + # + # should raise no exceptions: + self.assertEqual(self.__action.replaceTag('test=<A>', cm), "test=0") + # **Important**: recursive replacement of dynamic data from calling map should be prohibited, + # otherwise may be vulnerable on foreign user-input: + self.assertEqual(self.__action.replaceTag('test=<A>--<B>--<A>', cm), "test=0--<A><A>--0") + # should raise an exception (BOOM by replacement of tag "C"): + self.assertRaises(ZeroDivisionError, lambda: self.__action.replaceTag('test=<C>', cm)) + # should raise no exceptions (replaces tag "D" only): + self.assertEqual(self.__action.replaceTag('<D>', cm), "test=<C>") + def testReplaceTag(self): aInfo = { 'HOST': "192.0.2.0", @@ -186,7 +225,7 @@ class CommandActionTest(LogCaptureTestCase): # Callable self.assertEqual( self.__action.replaceTag("09 <matches> 11", - CallingMap(matches=lambda: str(10))), + CallingMap(matches=lambda self: str(10))), "09 10 11") def testReplaceNoTag(self): @@ -194,7 +233,27 @@ class CommandActionTest(LogCaptureTestCase): # Will raise ValueError if it is self.assertEqual( self.__action.replaceTag("abc", - CallingMap(matches=lambda: int("a"))), "abc") + CallingMap(matches=lambda self: int("a"))), "abc") + + def testReplaceTagSelfRecursion(self): + setattr(self.__action, 'a', "<a") + setattr(self.__action, 'b', "c>") + setattr(self.__action, 'b?family=inet6', "b>") + setattr(self.__action, 'ac', "<a><b>") + setattr(self.__action, 'ab', "<ac>") + setattr(self.__action, 'x?family=inet6', "") + # produce self-referencing properties except: + self.assertRaisesRegexp(ValueError, r"properties contain self referencing definitions", + lambda: self.__action.replaceTag("<a><b>", + self.__action._properties, conditional="family=inet4") + ) + # remore self-referencing in props: + delattr(self.__action, 'ac') + # produce self-referencing query except: + self.assertRaisesRegexp(ValueError, r"possible self referencing definitions in query", + lambda: self.__action.replaceTag("<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x>>>>>>>>>>>>>>>>>>>>>", + self.__action._properties, conditional="family=inet6") + ) def testReplaceTagConditionalCached(self): setattr(self.__action, 'abc', "123") @@ -217,10 +276,10 @@ class CommandActionTest(LogCaptureTestCase): self.__action.replaceTag("<banaction> '<abc>'", self.__action._properties, conditional="family=inet6", cache=cache), "Text 890-567 text 567 '567'") - self.assertEqual(len(cache) if cache is not None else -1, 3) + self.assertTrue(len(cache) >= 3) # set one parameter - internal properties and cache should be reseted: setattr(self.__action, 'xyz', "000-<abc>") - self.assertEqual(len(cache) if cache is not None else -1, 0) + self.assertEqual(len(cache), 0) # test againg, should have 000 instead of 890: for i in range(2): self.assertEqual( @@ -235,7 +294,7 @@ class CommandActionTest(LogCaptureTestCase): self.__action.replaceTag("<banaction> '<abc>'", self.__action._properties, conditional="family=inet6", cache=cache), "Text 000-567 text 567 '567'") - self.assertEqual(len(cache), 3) + self.assertTrue(len(cache) >= 3) def testExecuteActionBan(self): @@ -301,13 +360,24 @@ class CommandActionTest(LogCaptureTestCase): self.assertEqual(self.__action.ROST,"192.0.2.0") def testExecuteActionUnbanAinfo(self): - aInfo = { + aInfo = CallingMap({ 'ABC': "123", - } - self.__action.actionban = "touch /tmp/fail2ban.test.123" - self.__action.actionunban = "rm /tmp/fail2ban.test.<ABC>" + 'ip': '192.0.2.1', + 'F-*': lambda self: { + 'fid': 111, + 'fport': 222, + 'user': "tester" + } + }) + self.__action.actionban = "touch /tmp/fail2ban.test.123; echo 'failure <F-ID> of <F-USER> -<F-TEST>- from <ip>:<F-PORT>'" + self.__action.actionunban = "rm /tmp/fail2ban.test.<ABC>; echo 'user <F-USER> unbanned'" self.__action.ban(aInfo) self.__action.unban(aInfo) + self.assertLogged( + " -- stdout: 'failure 111 of tester -- from 192.0.2.1:222'", + " -- stdout: 'user tester unbanned'", + all=True + ) def testExecuteActionStartEmpty(self): self.__action.actionstart = "" @@ -319,6 +389,51 @@ class CommandActionTest(LogCaptureTestCase): self.assertLogged('Nothing to do') self.pruneLog() + def testExecuteWithVars(self): + self.assertTrue(self.__action.executeCmd( + r'''printf %b "foreign input:\n''' + r''' -- $f2bV_A --\n''' + r''' -- $f2bV_B --\n''' + r''' -- $(echo -n $f2bV_C) --''' # echo just replaces \n to test it as single line + r'''"''', + varsDict={ + 'f2bV_A': 'I\'m a hacker; && $(echo $f2bV_B)', + 'f2bV_B': 'I"m very bad hacker', + 'f2bV_C': '`Very | very\n$(bad & worst hacker)`' + })) + self.assertLogged(r"""foreign input:""", + ' -- I\'m a hacker; && $(echo $f2bV_B) --', + ' -- I"m very bad hacker --', + ' -- `Very | very $(bad & worst hacker)` --', all=True) + + def testExecuteReplaceEscapeWithVars(self): + self.__action.actionban = 'echo "** ban <ip>, reason: <reason> ...\\n<matches>"' + self.__action.actionunban = 'echo "** unban <ip>"' + self.__action.actionstop = 'echo "** stop monitoring"' + matches = [ + '<actionunban>', + '" Hooray! #', + '`I\'m cool script kiddy', + '`I`m very cool > /here-is-the-path/to/bin/.x-attempt.sh', + '<actionstop>', + ] + aInfo = { + 'ip': '192.0.2.1', + 'reason': 'hacking attempt ( he thought he knows how f2b internally works ;)', + 'matches': '\n'.join(matches) + } + self.pruneLog() + self.__action.ban(aInfo) + self.assertLogged( + '** ban %s' % aInfo['ip'], aInfo['reason'], *matches, all=True) + self.assertNotLogged( + '** unban %s' % aInfo['ip'], '** stop monitoring', all=True) + self.pruneLog() + self.__action.unban(aInfo) + self.__action.stop() + self.assertLogged( + '** unban %s' % aInfo['ip'], '** stop monitoring', all=True) + def testExecuteIncorrectCmd(self): CommandAction.executeCmd('/bin/ls >/dev/null\nbogusXXX now 2>/dev/null') self.assertLogged('HINT on 127: "Command not found"') @@ -330,8 +445,9 @@ class CommandActionTest(LogCaptureTestCase): self.assertFalse(CommandAction.executeCmd('sleep 30', timeout=timeout)) # give a test still 1 second, because system could be too busy self.assertTrue(time.time() >= stime + timeout and time.time() <= stime + timeout + 1) - self.assertLogged('sleep 30 -- timed out after') - self.assertLogged('sleep 30 -- killed with SIGTERM') + self.assertLogged('sleep 30', ' -- timed out after', all=True) + self.assertLogged(' -- killed with SIGTERM', + ' -- killed with SIGKILL') def testExecuteTimeoutWithNastyChildren(self): # temporary file for a nasty kid shell script @@ -387,9 +503,9 @@ class CommandActionTest(LogCaptureTestCase): # Verify that the process itself got killed self.assertTrue(Utils.wait_for(lambda: not pid_exists(cpid), 3)) self.assertLogged('my pid ', 'Resource temporarily unavailable') - self.assertLogged('timed out') - self.assertLogged('killed with SIGTERM', - 'killed with SIGKILL') + self.assertLogged(' -- timed out') + self.assertLogged(' -- killed with SIGTERM', + ' -- killed with SIGKILL') os.unlink(tmpFilename) os.unlink(tmpFilename + '.pid') @@ -403,7 +519,7 @@ class CommandActionTest(LogCaptureTestCase): "stderr: 'The rain in Spain stays mainly in the plain'\n") def testCallingMap(self): - mymap = CallingMap(callme=lambda: str(10), error=lambda: int('a'), + mymap = CallingMap(callme=lambda self: str(10), error=lambda self: int('a'), dontcallme= "string", number=17) # Should work fine @@ -412,3 +528,43 @@ class CommandActionTest(LogCaptureTestCase): "10 okay string 17") # Error will now trip, demonstrating delayed call self.assertRaises(ValueError, lambda x: "%(error)i" % x, mymap) + + def testCallingMapModify(self): + m = CallingMap({ + 'a': lambda self: 2 + 3, + 'b': lambda self: self['a'] + 6, + 'c': 'test', + }) + # test reset (without modifications): + m.reset() + # do modifications: + m['a'] = 4 + del m['c'] + # test set and delete: + self.assertEqual(len(m), 2) + self.assertNotIn('c', m) + self.assertEqual((m['a'], m['b']), (4, 10)) + # reset to original and test again: + m.reset() + s = repr(m) + self.assertEqual(len(m), 3) + self.assertIn('c', m) + self.assertEqual((m['a'], m['b'], m['c']), (5, 11, 'test')) + + def testCallingMapRep(self): + m = CallingMap({ + 'a': lambda self: 2 + 3, + 'b': lambda self: self['a'] + 6, + 'c': '' + }) + s = repr(m) + self.assertIn("'a': 5", s) + self.assertIn("'b': 11", s) + self.assertIn("'c': ''", s) + + m['c'] = lambda self: self['xxx'] + 7; # unresolvable + s = repr(m) + self.assertIn("'a': 5", s) + self.assertIn("'b': 11", s) + self.assertIn("'c': ", s) # presents as callable + self.assertNotIn("'c': ''", s) # but not empty diff --git a/fail2ban/tests/banmanagertestcase.py b/fail2ban/tests/banmanagertestcase.py index 2c9d6601..33cd2dac 100644 --- a/fail2ban/tests/banmanagertestcase.py +++ b/fail2ban/tests/banmanagertestcase.py @@ -38,7 +38,7 @@ class AddFailure(unittest.TestCase): def tearDown(self): """Call after every test case.""" - pass + super(AddFailure, self).tearDown() def testAdd(self): self.assertTrue(self.__banManager.addBanTicket(self.__ticket)) @@ -147,7 +147,7 @@ class StatusExtendedCymruInfo(unittest.TestCase): def tearDown(self): """Call after every test case.""" - pass + super(StatusExtendedCymruInfo, self).tearDown() available = True, None @@ -198,13 +198,12 @@ class StatusExtendedCymruInfo(unittest.TestCase): "country": ["nxdomain"], "rir": ["nxdomain"]}) - # even for private IPs ASNs defined # Since it outputs for all active tickets we would get previous results # and new ones - ticket = BanTicket("10.0.0.0", 1167606000.0) + ticket = BanTicket("8.0.0.0", 1167606000.0) self.assertTrue(self.__banManager.addBanTicket(ticket)) cymru_info = self._getBanListExtendedCymruInfo() self.assertDictEqual(dict((k, sorted(v)) for k, v in cymru_info.iteritems()), - {"asn": sorted(["nxdomain", "4565",]), - "country": sorted(["nxdomain", "unknown"]), - "rir": sorted(["nxdomain", "other"])}) + {"asn": sorted(["nxdomain", "3356",]), + "country": sorted(["nxdomain", "US"]), + "rir": sorted(["nxdomain", "arin"])}) diff --git a/fail2ban/tests/clientbeautifiertestcase.py b/fail2ban/tests/clientbeautifiertestcase.py index 7d20e84e..79a0ff54 100644 --- a/fail2ban/tests/clientbeautifiertestcase.py +++ b/fail2ban/tests/clientbeautifiertestcase.py @@ -37,6 +37,7 @@ class BeautifierTest(unittest.TestCase): def tearDown(self): """ Call after every test case """ + super(BeautifierTest, self).tearDown() def testGetInputCmd(self): cmd = ["test"] diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 39622cd0..c3a10c36 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -28,7 +28,7 @@ import re import shutil import tempfile import unittest -from ..client.configreader import ConfigReader, ConfigReaderUnshared +from ..client.configreader import ConfigReader, ConfigReaderUnshared, NoSectionError from ..client import configparserinc from ..client.jailreader import JailReader from ..client.filterreader import FilterReader @@ -162,6 +162,40 @@ c = d ;in line comment self.assertEqual(self.c.get('DEFAULT', 'b'), 'a') self.assertEqual(self.c.get('DEFAULT', 'c'), 'd') + def testTargetedSectionOptions(self): + self.assertFalse(self.c.read('g')) # nothing is there yet + self._write("g.conf", value=None, content=""" +[DEFAULT] +a = def-a +b = def-b,a:`%(a)s` +c = def-c,b:"%(b)s" +d = def-d-b:"%(known/b)s" + +[jail] +a = jail-a-%(test/a)s +b = jail-b-%(test/b)s +y = %(test/y)s + +[test] +a = test-a-%(default/a)s +b = test-b-%(known/b)s +x = %(test/x)s +y = %(jail/y)s +""") + self.assertTrue(self.c.read('g')) + self.assertEqual(self.c.get('test', 'a'), 'test-a-def-a') + self.assertEqual(self.c.get('test', 'b'), 'test-b-def-b,a:`test-a-def-a`') + self.assertEqual(self.c.get('jail', 'a'), 'jail-a-test-a-def-a') + self.assertEqual(self.c.get('jail', 'b'), 'jail-b-test-b-def-b,a:`jail-a-test-a-def-a`') + self.assertEqual(self.c.get('jail', 'c'), 'def-c,b:"jail-b-test-b-def-b,a:`jail-a-test-a-def-a`"') + self.assertEqual(self.c.get('jail', 'd'), 'def-d-b:"def-b,a:`jail-a-test-a-def-a`"') + self.assertEqual(self.c.get('test', 'c'), 'def-c,b:"test-b-def-b,a:`test-a-def-a`"') + self.assertEqual(self.c.get('test', 'd'), 'def-d-b:"def-b,a:`test-a-def-a`"') + self.assertEqual(self.c.get('DEFAULT', 'c'), 'def-c,b:"def-b,a:`def-a`"') + self.assertEqual(self.c.get('DEFAULT', 'd'), 'def-d-b:"def-b,a:`def-a`"') + self.assertRaises(Exception, self.c.get, 'test', 'x') + self.assertRaises(Exception, self.c.get, 'jail', 'y') + class JailReaderTest(LogCaptureTestCase): @@ -196,6 +230,14 @@ class JailReaderTest(LogCaptureTestCase): self.assertTrue(jail.isEnabled()) self.assertLogged("Invalid action definition 'joho[foo'") + def testJailLogTimeZone(self): + jail = JailReader('tz_correct', basedir=IMPERFECT_CONFIG, + share_config=IMPERFECT_CONFIG_SHARE_CFG) + self.assertTrue(jail.read()) + self.assertTrue(jail.getOptions()) + self.assertTrue(jail.isEnabled()) + self.assertEqual(jail.options['logtimezone'], 'UTC+0200') + def testJailFilterBrokenDef(self): jail = JailReader('brokenfilterdef', basedir=IMPERFECT_CONFIG, share_config=IMPERFECT_CONFIG_SHARE_CFG) @@ -317,7 +359,17 @@ class JailReaderTest(LogCaptureTestCase): self.assertLogged('File %s is a dangling link, thus cannot be monitored' % f2) self.assertEqual(JailReader._glob(os.path.join(d, 'nonexisting')), []) - + def testCommonFunction(self): + c = ConfigReader(share_config={}) + # test common functionalities (no shared, without read of config): + self.assertEqual(c.sections(), []) + self.assertFalse(c.has_section('test')) + self.assertRaises(NoSectionError, c.merge_section, 'test', {}) + self.assertRaises(NoSectionError, c.options, 'test') + self.assertRaises(NoSectionError, c.get, 'test', 'any') + self.assertRaises(NoSectionError, c.getOptions, 'test', {}) + + class FilterReaderTest(unittest.TestCase): def __init__(self, *args, **kwargs): @@ -347,7 +399,7 @@ class FilterReaderTest(unittest.TestCase): ['set', 'testcase01', 'addjournalmatch', "FIELD= with spaces ", "+", "AFIELD= with + char and spaces"], ['set', 'testcase01', 'datepattern', "%Y %m %d %H:%M:%S"], - ['set', 'testcase01', 'maxlines', "1"], # Last for overide test + ['set', 'testcase01', 'maxlines', 1], # Last for overide test ] filterReader = FilterReader("testcase01", "testcase01", {}) filterReader.setBaseDir(TEST_FILES_DIR) @@ -517,18 +569,20 @@ class JailsReaderTest(LogCaptureTestCase): ['add', 'brokenaction', 'auto'], ['set', 'brokenaction', 'addfailregex', '<IP>'], ['set', 'brokenaction', 'addaction', 'brokenaction'], - ['set', - 'brokenaction', - 'action', - 'brokenaction', - 'actionban', - 'hit with big stick <ip>'], + ['multi-set', 'brokenaction', 'action', 'brokenaction', [ + ['actionban', 'hit with big stick <ip>'], + ['actname', 'brokenaction'] + ]], ['add', 'parse_to_end_of_jail.conf', 'auto'], ['set', 'parse_to_end_of_jail.conf', 'addfailregex', '<IP>'], + ['set', 'tz_correct', 'addfailregex', '<IP>'], + ['set', 'tz_correct', 'logtimezone', 'UTC+0200'], ['start', 'emptyaction'], ['start', 'missinglogfiles'], ['start', 'brokenaction'], ['start', 'parse_to_end_of_jail.conf'], + ['add', 'tz_correct', 'auto'], + ['start', 'tz_correct'], ['config-error', "Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo'"], ['config-error', @@ -548,7 +602,10 @@ class JailsReaderTest(LogCaptureTestCase): actionName = os.path.basename(actionConfig).replace('.conf', '') actionReader = ActionReader(actionName, "TEST", {}, basedir=CONFIG_DIR) self.assertTrue(actionReader.read()) - actionReader.getOptions({}) # populate _opts + try: + actionReader.getOptions({}) # populate _opts + except Exception as e: # pragma: no cover + self.fail("action %r\n%s: %s" % (actionName, type(e).__name__, e)) if not actionName.endswith('-common'): self.assertIn('Definition', actionReader.sections(), msg="Action file %r is lacking [Definition] section" % actionConfig) @@ -627,7 +684,7 @@ class JailsReaderTest(LogCaptureTestCase): # grab all filter names filters = set(os.path.splitext(os.path.split(a)[1])[0] for a in glob.glob(os.path.join('config', 'filter.d', '*.conf')) - if not a.endswith('common.conf')) + if not (a.endswith('common.conf') or a.endswith('-aggressive.conf'))) # get filters of all jails (filter names without options inside filter[...]) filters_jail = set( JailReader.extractOptions(jail.options['filter'])[0] for jail in jails.jails @@ -711,6 +768,7 @@ class JailsReaderTest(LogCaptureTestCase): self.assertEqual(opts['socket'], '/var/run/fail2ban/fail2ban.sock') self.assertEqual(opts['pidfile'], '/var/run/fail2ban/fail2ban.pid') + configurator.readAll() configurator.getOptions() configurator.convertToProtocol() commands = configurator.getConfigStream() diff --git a/fail2ban/tests/config/filter.d/common.conf b/fail2ban/tests/config/filter.d/common.conf deleted file mode 120000 index 83e92474..00000000 --- a/fail2ban/tests/config/filter.d/common.conf +++ /dev/null @@ -1 +0,0 @@ -../../../../config/filter.d/common.conf
\ No newline at end of file diff --git a/fail2ban/tests/config/filter.d/test.conf b/fail2ban/tests/config/filter.d/test.conf index f09d3467..9d08ef09 100644 --- a/fail2ban/tests/config/filter.d/test.conf +++ b/fail2ban/tests/config/filter.d/test.conf @@ -1,6 +1,13 @@ #[INCLUDES] #before = common.conf +[DEFAULT] +_daemon = default + [Definition] -failregex = failure test 1 (filter.d/test.conf) <HOST> +where = conf +failregex = failure <_daemon> <one> (filter.d/test.%(where)s) <HOST> +[Init] +# test parameter, should be overriden in jail by "filter=test[one=1,...]" +one = *1* diff --git a/fail2ban/tests/config/filter.d/test.local b/fail2ban/tests/config/filter.d/test.local index 1b6cf55e..a954f81e 100644 --- a/fail2ban/tests/config/filter.d/test.local +++ b/fail2ban/tests/config/filter.d/test.local @@ -2,6 +2,15 @@ #before = common.conf [Definition] +# overwrite default daemon, additionally it should be accessible in jail with "%(known/_daemon)s": +_daemon = test +# interpolate previous regex (from test.conf) + new 2nd + dynamical substitution) of "two" an "where": failregex = %(known/failregex)s - failure test 2 (filter.d/test.local) <HOST> + failure %(_daemon)s <two> (filter.d/test.<where>) <HOST> +# parameter "two" should be specified in jail by "filter=test[..., two=2]" +[Init] +# this parameter can be used in jail with "%(known/three)s": +three = 3 +# this parameter "where" does not overwrite "where" in definition of test.conf (dynamical values only): +where = local
\ No newline at end of file diff --git a/fail2ban/tests/config/filter.d/zzz-generic-example.conf b/fail2ban/tests/config/filter.d/zzz-generic-example.conf index 8d1f2ae9..c50f4dc6 100644 --- a/fail2ban/tests/config/filter.d/zzz-generic-example.conf +++ b/fail2ban/tests/config/filter.d/zzz-generic-example.conf @@ -8,7 +8,7 @@ # Read common prefixes. If any customizations available -- read them from # common.local. common.conf is a symlink to the original common.conf and # should be copied (dereferenced) during installation -before = common.conf +before = ../../../../config/filter.d/common.conf [Definition] diff --git a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf new file mode 100644 index 00000000..d6eecd4b --- /dev/null +++ b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf @@ -0,0 +1,89 @@ +# Fail2Ban obsolete multiline example resp. test filter (previously sshd.conf) +# + +[INCLUDES] + +# Read common prefixes. If any customizations available -- read them from +# common.local +before = ../../../../config/filter.d/common.conf + +[DEFAULT] + +_daemon = sshd + +# optional prefix (logged from several ssh versions) like "error: ", "error: PAM: " or "fatal: " +__pref = (?:(?:error|fatal): (?:PAM: )?)? +# optional suffix (logged from several ssh versions) like " [preauth]" +__suff = (?: \[preauth\])?\s* +__on_port_opt = (?: port \d+)?(?: on \S+(?: port \d+)?)? + +# single line prefix: +__prefix_line_sl = %(__prefix_line)s%(__pref)s +# multi line prefixes (for first and second lines): +__prefix_line_ml1 = (?P<__prefix>%(__prefix_line)s)%(__pref)s +__prefix_line_ml2 = %(__suff)s$<SKIPLINES>^(?P=__prefix)%(__pref)s + +[Definition] + +cmnfailre = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for .* from <HOST>( via \S+)?\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser not known to the underlying authentication module for .* from <HOST>\s*%(__suff)s$ + ^%(__prefix_line_sl)sFailed \S+ for (?P<cond_inv>invalid user )?(?P<user>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)) from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$) + ^%(__prefix_line_sl)sROOT LOGIN REFUSED.* FROM <HOST>\s*%(__suff)s$ + ^%(__prefix_line_sl)s[iI](?:llegal|nvalid) user .*? from <HOST>%(__on_port_opt)s\s*$ + ^%(__prefix_line_sl)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from <HOST> not allowed because listed in DenyUsers\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from <HOST> not allowed because not in any group\s*%(__suff)s$ + ^%(__prefix_line_sl)srefused connect from \S+ \(<HOST>\)\s*%(__suff)s$ + ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from <HOST> not allowed because a group is listed in DenyGroups\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$ + ^%(__prefix_line_sl)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=<HOST>\s.*%(__suff)s$ + ^%(__prefix_line_sl)s(error: )?maximum authentication attempts exceeded for .* from <HOST>%(__on_port_opt)s(?: ssh\d*)? \[preauth\]$ + ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from <HOST>: 11: .+%(__suff)s$ + ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s%(__prefix_line_ml2)sConnection closed by <HOST>%(__suff)s$ + ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures(?: for .+?)?%(__suff)s$ + +mdre-normal = + +mdre-ddos = ^%(__prefix_line_sl)sDid not receive identification string from <HOST>%(__suff)s$ + ^%(__prefix_line_sl)sConnection reset by <HOST>%(__on_port_opt)s%(__suff)s + ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$ + +mdre-extra = ^%(__prefix_line_sl)sReceived disconnect from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$ + ^%(__prefix_line_sl)sUnable to negotiate with <HOST>%(__on_port_opt)s: no matching (?:cipher|key exchange method) found. + ^%(__prefix_line_ml1)sConnection from <HOST>%(__on_port_opt)s%(__prefix_line_ml2)sUnable to negotiate a (?:cipher|key exchange method)%(__suff)s$ + +mdre-aggressive = %(mdre-ddos)s + %(mdre-extra)s + +failregex = %(cmnfailre)s + <mdre-<mode>> + +# Parameter "mode": normal (default), ddos, extra or aggressive (combines all) +# Usage example (for jail.local): +# [sshd] +# mode = extra +# # or another jail (rewrite filter parameters of jail): +# [sshd-aggressive] +# filter = sshd[mode=aggressive] +# +mode = normal + +ignoreregex = + +# "maxlines" is number of log lines to buffer for multi-line regex searches +maxlines = 10 + +journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd + +datepattern = {^LN-BEG} + +# DEV Notes: +# +# "Failed \S+ for .*? from <HOST>..." failregex uses non-greedy catch-all because +# it is coming before use of <HOST> which is not hard-anchored at the end as well, +# and later catch-all's could contain user-provided input, which need to be greedily +# matched away first. +# +# Author: Cyril Jaquier, Yaroslav Halchenko, Petr Voralek, Daniel Black + diff --git a/fail2ban/tests/config/jail.conf b/fail2ban/tests/config/jail.conf index 659e3fd3..3dcbf634 100644 --- a/fail2ban/tests/config/jail.conf +++ b/fail2ban/tests/config/jail.conf @@ -15,9 +15,9 @@ ignoreip = [test-known-interp] enabled = true -filter = test +filter = test[one=1,two=2] failregex = %(known/failregex)s - failure test 3 (jail.local) <HOST> + failure %(known/_daemon)s %(known/three)s (jail.local) <HOST> [missinglogfiles] enabled = true @@ -47,3 +47,7 @@ action = thefunkychickendance [parse_to_end_of_jail.conf] enabled = true action = + +[tz_correct] +enabled = true +logtimezone = UTC+0200 diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index b1c68a8f..cbfc1517 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -358,6 +358,19 @@ class DatabaseTest(LogCaptureTestCase): self.assertEqual(len(tickets), 2) ticket = self.db.getCurrentBans(jail=None, ip="127.0.0.1"); self.assertEqual(ticket.getIP(), "127.0.0.1") + + # positive case (1 ticket not yet expired): + tickets = self.db.getCurrentBans(jail=self.jail, forbantime=15, + fromtime=MyTime.time()) + self.assertEqual(len(tickets), 1) + # negative case (all are expired in 1year): + tickets = self.db.getCurrentBans(jail=self.jail, forbantime=15, + fromtime=MyTime.time() + MyTime.str2seconds("1year")) + self.assertEqual(len(tickets), 0) + # persistent bantime (-1), so never expired: + tickets = self.db.getCurrentBans(jail=self.jail, forbantime=-1, + fromtime=MyTime.time() + MyTime.str2seconds("1year")) + self.assertEqual(len(tickets), 2) def testActionWithDB(self): # test action together with database functionality diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index 5b32a7e9..02facf30 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -89,6 +89,55 @@ class DateDetectorTest(LogCaptureTestCase): self.assertEqual(datelog, dateUnix) self.assertEqual(matchlog.group(1), 'Jan 23 21:59:59') + def testDefaultTimeZone(self): + # use special date-pattern (with %Exz), because %z currently does not supported + # zone abbreviations except Z|UTC|GMT. + dd = DateDetector() + dd.appendTemplate('^%ExY-%Exm-%Exd %H:%M:%S(?: ?%Exz)?') + dt = datetime.datetime + logdt = "2017-01-23 15:00:00" + dtUTC = dt(2017, 1, 23, 15, 0) + for tz, log, desired in ( + # no TZ in input-string: + ('UTC+0300', logdt, dt(2017, 1, 23, 12, 0)), # so in UTC, it was noon + ('UTC', logdt, dtUTC), # UTC + ('UTC-0430', logdt, dt(2017, 1, 23, 19, 30)), + ('GMT+12', logdt, dt(2017, 1, 23, 3, 0)), + (None, logdt, dt(2017, 1, 23, 14, 0)), # default CET in our test-framework + # CET: + ('CET', logdt, dt(2017, 1, 23, 14, 0)), + ('+0100', logdt, dt(2017, 1, 23, 14, 0)), + ('CEST-01', logdt, dt(2017, 1, 23, 14, 0)), + # CEST: + ('CEST', logdt, dt(2017, 1, 23, 13, 0)), + ('+0200', logdt, dt(2017, 1, 23, 13, 0)), + ('CET+01', logdt, dt(2017, 1, 23, 13, 0)), + ('CET+0100', logdt, dt(2017, 1, 23, 13, 0)), + # check offset in minutes: + ('CET+0130', logdt, dt(2017, 1, 23, 12, 30)), + # TZ in input-string have precedence: + ('UTC+0300', logdt+' GMT', dtUTC), # GMT wins + ('UTC', logdt+' GMT', dtUTC), # GMT wins + ('UTC-0430', logdt+' GMT', dtUTC), # GMT wins + (None, logdt+' GMT', dtUTC), # GMT wins + ('UTC', logdt+' -1045', dt(2017, 1, 24, 1, 45)), # -1045 wins + (None, logdt+' -10:45', dt(2017, 1, 24, 1, 45)), # -1045 wins + ('UTC', logdt+' +0945', dt(2017, 1, 23, 5, 15)), # +0945 wins + (None, logdt+' +09:45', dt(2017, 1, 23, 5, 15)), # +0945 wins + ('UTC+0300', logdt+' Z', dtUTC), # Z wins (UTC) + ('GMT+12', logdt+' CET', dt(2017, 1, 23, 14, 0)), # CET wins + ('GMT+12', logdt+' CEST', dt(2017, 1, 23, 13, 0)), # CEST wins + ('GMT+12', logdt+' CET+0130', dt(2017, 1, 23, 12, 30)), # CET+0130 wins + ): + logSys.debug('== test %r with TZ %r', log, tz) + dd.default_tz=tz; datelog, _ = dd.getTime(log) + val = dt.utcfromtimestamp(datelog) + self.assertEqual(val, desired, + "wrong offset %r != %r by %r with default TZ %r (%r)" % (val, desired, log, tz, dd.default_tz)) + + self.assertRaises(ValueError, setattr, dd, 'default_tz', 'WRONG-TZ') + dd.default_tz = None + def testVariousTimes(self): """Test detection of various common date/time formats f2b should understand """ @@ -298,6 +347,16 @@ iso8601 = DatePatternRegex("%Y-%m-%d[T ]%H:%M:%S(?:\.%f)?%z") class CustomDateFormatsTest(unittest.TestCase): + def setUp(self): + """Call before every test case.""" + unittest.TestCase.setUp(self) + setUpMyTime() + + def tearDown(self): + """Call after every test case.""" + unittest.TestCase.tearDown(self) + tearDownMyTime() + def testIso8601(self): date = datetime.datetime.utcfromtimestamp( iso8601.getDate("2007-01-25T12:00:00Z")[0]) @@ -411,6 +470,37 @@ class CustomDateFormatsTest(unittest.TestCase): else: self.assertEqual(date, None) + def testVariousFormatSpecs(self): + for (matched, dp, line) in ( + # cover %B (full-month-name) and %I (as 12 == 0): + (1106438399.0, "^%B %Exd %I:%ExM:%ExS**", 'January 23 12:59:59'), + # cover %U (week of year starts on sunday) and %A (weekday): + (985208399.0, "^%y %U %A %ExH:%ExM:%ExS**", '01 11 Wednesday 21:59:59'), + # cover %W (week of year starts on monday) and %A (weekday): + (984603599.0, "^%y %W %A %ExH:%ExM:%ExS**", '01 11 Wednesday 21:59:59'), + # cover %W (week of year starts on monday) and %w (weekday, 0 - sunday): + (984949199.0, "^%y %W %w %ExH:%ExM:%ExS**", '01 11 0 21:59:59'), + # cover %W (week of year starts on monday) and %w (weekday, 6 - saturday): + (984862799.0, "^%y %W %w %ExH:%ExM:%ExS**", '01 11 6 21:59:59'), + # cover time only, current date, in test cases now == 14 Aug 2005 12:00 -> back to yesterday (13 Aug): + (1123963199.0, "^%ExH:%ExM:%ExS**", '21:59:59'), + # cover time only, current date, in test cases now == 14 Aug 2005 12:00 -> today (14 Aug): + (1123970401.0, "^%ExH:%ExM:%ExS**", '00:00:01'), + # cover date with current year, in test cases now == Aug 2005 -> back to last year (Sep 2004): + (1094068799.0, "^%m/%d %ExH:%ExM:%ExS**", '09/01 21:59:59'), + ): + logSys.debug('== test: %r', (matched, dp, line)) + dd = DateDetector() + dd.appendTemplate(dp) + date = dd.getTime(line) + if matched: + self.assertTrue(date) + if isinstance(matched, basestring): # pragma: no cover + self.assertEqual(matched, date[1].group(1)) + else: + self.assertEqual(matched, date[0]) + else: # pragma: no cover + self.assertEqual(date, None) # def testDefaultTempate(self): # self.__datedetector.setDefaultRegex("^\S{3}\s{1,2}\d{1,2} \d{2}:\d{2}:\d{2}") diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index e6cc46cb..caacf63c 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -416,6 +416,9 @@ class Fail2banClientTest(Fail2banClientServerBase): self.pruneLog() self.execSuccess((), "-vq", "-V") self.assertLogged("Fail2Ban v" + fail2bancmdline.version) + self.pruneLog() + self.execSuccess((), "--str2sec", "1d12h30m") + self.assertLogged("131400") @with_tmpdir def testClientDump(self, tmp): @@ -755,12 +758,19 @@ class Fail2banServerTest(Fail2banClientServerBase): os.remove(fn) return _write_file(fn, "w", + "[DEFAULT]", + "_exec_once = 0", + "", "[Definition]", - "actionstart = echo '[<name>] %s: ** start'" % actname, start, - "actionreload = echo '[<name>] %s: .. reload'" % actname, reload, - "actionban = echo '[<name>] %s: ++ ban <ip>'" % actname, ban, - "actionunban = echo '[<name>] %s: -- unban <ip>'" % actname, unban, - "actionstop = echo '[<name>] %s: __ stop'" % actname, stop, + "norestored = %(_exec_once)s", + "restore = ", + "info = ", + "_use_flush_ = echo [<name>] <actname>: -- flushing IPs", + "actionstart = echo '[%(name)s] %(actname)s: ** start'", start, + "actionreload = echo '[%(name)s] %(actname)s: .. reload'", reload, + "actionban = echo '[%(name)s] %(actname)s: ++ ban <ip> %(restore)s%(info)s'", ban, + "actionunban = echo '[%(name)s] %(actname)s: -- unban <ip>'", unban, + "actionstop = echo '[%(name)s] %(actname)s: __ stop'", stop, ) if unittest.F2B.log_level <= logging.DEBUG: # pragma: no cover _out_file(fn) @@ -772,22 +782,34 @@ class Fail2banServerTest(Fail2banClientServerBase): "usedns = no", "maxretry = 3", "findtime = 10m", - "failregex = ^\s*failure (401|403) from <HOST>", + "failregex = ^\s*failure <F-ERRCODE>401|403</F-ERRCODE> from <HOST>", "datepattern = {^LN-BEG}EPOCH", + "ignoreip = 127.0.0.1/8 ::1", # just to cover ignoreip in jailreader/transmitter "", "[test-jail1]", "backend = " + backend, "filter =", "action = ", - " test-action1[name='%(__name__)s']" if 1 in actions else "", - " test-action2[name='%(__name__)s']" if 2 in actions else "", + " test-action1[name='%(__name__)s']" \ + if 1 in actions else "", + " test-action2[name='%(__name__)s', restore='restored: <restored>', info=', err-code: <F-ERRCODE>']" \ + if 2 in actions else "", + " test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>'," + " actionflush=<_use_flush_>]" \ + if 3 in actions else "", "logpath = " + test1log, " " + test2log if 2 in enabled else "", " " + test3log if 2 in enabled else "", - "failregex = ^\s*failure (401|403) from <HOST>", - " ^\s*error (401|403) from <HOST>" if 2 in enabled else "", + "failregex = ^\s*failure <F-ERRCODE>401|403</F-ERRCODE> from <HOST>", + " ^\s*error <F-ERRCODE>401|403</F-ERRCODE> from <HOST>" \ + if 2 in enabled else "", "enabled = true" if 1 in enabled else "", "", "[test-jail2]", "backend = " + backend, "filter =", - "action =", + "action = ", + " test-action2[name='%(__name__)s', restore='restored: <restored>', info=', err-code: <F-ERRCODE>']" \ + if 2 in actions else "", + " test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: <restored>']" + " actionflush=<_use_flush_>]" \ + if 3 in actions else "", "logpath = " + test2log, "enabled = true" if 2 in enabled else "", ) @@ -798,7 +820,7 @@ class Fail2banServerTest(Fail2banClientServerBase): _write_action_cfg(actname="test-action1") _write_action_cfg(actname="test-action2") - _write_jail_cfg(enabled=[1], actions=[1,2]) + _write_jail_cfg(enabled=[1], actions=[1,2,3]) # append one wrong configured jail: _write_file(pjoin(cfg, "jail.conf"), "a", "", "[broken-jail]", "", "filter = broken-jail-filter", "enabled = true") @@ -821,6 +843,11 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertLogged( "stdout: '[test-jail1] test-action1: ** start'", "stdout: '[test-jail1] test-action2: ** start'", all=True) + # test restored is 0 (both actions available): + self.assertLogged( + "stdout: '[test-jail1] test-action2: ++ ban 192.0.2.1 restored: 0, err-code: 401'", + "stdout: '[test-jail1] test-action3: ++ ban 192.0.2.1 restored: 0'", + all=True, wait=MID_WAITTIME) # broken jail was logged (in client and server log): self.assertLogged( @@ -853,6 +880,12 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertLogged( "Creating new jail 'test-jail2'", "Jail 'test-jail2' started", all=True) + # test action3 removed, test flushing successful (and no single unban occurred): + self.assertLogged( + "stdout: '[test-jail1] test-action3: -- flushing IPs'", + "stdout: '[test-jail1] test-action3: __ stop'", all=True) + self.assertNotLogged( + "stdout: '[test-jail1] test-action3: -- unban 192.0.2.1'") # update action1, delete action2 (should be stopped via configuration)... self.pruneLog("[test-phase 2a]") @@ -882,10 +915,10 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertNotLogged( "stdout: '[test-jail1] test-action1: -- unban 192.0.2.1'") - # don't need both actions anymore: + # don't need action1 anymore: _write_action_cfg(actname="test-action1", allow=False) - _write_action_cfg(actname="test-action2", allow=False) - _write_jail_cfg(actions=[]) + # leave action2 just to test restored interpolation: + _write_jail_cfg(actions=[2,3]) # write new failures: self.pruneLog("[test-phase 2b]") @@ -913,7 +946,8 @@ class Fail2banServerTest(Fail2banClientServerBase): "[test-jail2] Found 192.0.2.2", "[test-jail2] Ban 192.0.2.2", "[test-jail2] Found 192.0.2.3", - "[test-jail2] Ban 192.0.2.3", all=True) + "[test-jail2] Ban 192.0.2.3", + all=True) # rotate logs: _write_file(test1log, "w+") @@ -936,10 +970,30 @@ class Fail2banServerTest(Fail2banClientServerBase): "[test-jail2] Restore Ban 192.0.2.4", "[test-jail2] Restore Ban 192.0.2.8", all=True ) + # test restored is 1 (only test-action2): + self.assertLogged( + "stdout: '[test-jail2] test-action2: ++ ban 192.0.2.4 restored: 1, err-code: 401'", + "stdout: '[test-jail2] test-action2: ++ ban 192.0.2.8 restored: 1, err-code: 401'", + all=True, wait=MID_WAITTIME) + # test test-action3 not executed at all (norestored check): + self.assertNotLogged( + "stdout: '[test-jail2] test-action3: ++ ban 192.0.2.4 restored: 1'", + "stdout: '[test-jail2] test-action3: ++ ban 192.0.2.8 restored: 1'", + all=True) - # restart jail with unban all: + # ban manually to test later flush by unban all: self.pruneLog("[test-phase 2d]") self.execSuccess(startparams, + "set", "test-jail2", "banip", "192.0.2.21") + self.execSuccess(startparams, + "set", "test-jail2", "banip", "192.0.2.22") + self.assertLogged( + "stdout: '[test-jail2] test-action3: ++ ban 192.0.2.22", + "stdout: '[test-jail2] test-action3: ++ ban 192.0.2.22 ", all=True, wait=MID_WAITTIME) + + # restart jail with unban all: + self.pruneLog("[test-phase 2e]") + self.execSuccess(startparams, "restart", "--unban", "test-jail2") self.assertLogged( "Reload finished.", @@ -950,12 +1004,26 @@ class Fail2banServerTest(Fail2banClientServerBase): "[test-jail2] Unban 192.0.2.4", "[test-jail2] Unban 192.0.2.8", all=True ) + # test unban (action2): + self.assertLogged( + "stdout: '[test-jail2] test-action2: -- unban 192.0.2.21", + "stdout: '[test-jail2] test-action2: -- unban 192.0.2.22'", all=True) + # test flush (action3, and no single unban via action3 occurred): + self.assertLogged( + "stdout: '[test-jail2] test-action3: -- flushing IPs'") + self.assertNotLogged( + "stdout: '[test-jail2] test-action3: -- unban 192.0.2.21'", + "stdout: '[test-jail2] test-action3: -- unban 192.0.2.22'", all=True) # no more ban (unbanned all): self.assertNotLogged( "[test-jail2] Ban 192.0.2.4", "[test-jail2] Ban 192.0.2.8", all=True ) + # don't need actions anymore: + _write_action_cfg(actname="test-action2", allow=False) + _write_jail_cfg(actions=[]) + # reload jail1 without restart (without ban/unban): self.pruneLog("[test-phase 3]") self.execSuccess(startparams, "reload", "test-jail1") @@ -1040,6 +1108,14 @@ class Fail2banServerTest(Fail2banClientServerBase): "[test-jail1] Ban 192.0.2.4", all=True ) + # unban all (just to test command, already empty - nothing to unban): + self.pruneLog("[test-phase 7b]") + self.execSuccess(startparams, + "--async", "unban", "--all") + self.assertLogged( + "Flush ban list", + "Unbanned 0, 0 ticket(s) in 'test-jail1'", all=True) + # backend-switch (restart instead of reload): self.pruneLog("[test-phase 8a]") _write_jail_cfg(enabled=[1], backend="xxx-unknown-backend-zzz") diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 5767a3f7..f3a51773 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -27,17 +27,18 @@ import os import sys from ..client import fail2banregex -from ..client.fail2banregex import Fail2banRegex, get_opt_parser, exec_command_line, output +from ..client.fail2banregex import Fail2banRegex, get_opt_parser, exec_command_line, output, str2LogLevel from .utils import setUpMyTime, tearDownMyTime, LogCaptureTestCase, logSys from .utils import CONFIG_DIR fail2banregex.logSys = logSys def _test_output(*args): - logSys.info(args[0]) + logSys.notice(args[0]) fail2banregex.output = _test_output +TEST_CONFIG_DIR = os.path.join(os.path.dirname(__file__), "config") TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") DEV_NULL = None @@ -45,6 +46,9 @@ DEV_NULL = None def _Fail2banRegex(*args): parser = get_opt_parser() (opts, args) = parser.parse_args(list(args)) + # put down log-level if expected, because of too many debug-messages: + if opts.log_level in ("notice", "warning"): + logSys.setLevel(str2LogLevel(opts.log_level)) return (opts, args, Fail2banRegex(opts)) class ExitException(Exception): @@ -80,7 +84,13 @@ class Fail2banRegexTest(LogCaptureTestCase): FILENAME_02 = os.path.join(TEST_FILES_DIR, "testcase02.log") FILENAME_WRONGCHAR = os.path.join(TEST_FILES_DIR, "testcase-wrong-char.log") + FILENAME_SSHD = os.path.join(TEST_FILES_DIR, "logs", "sshd") FILTER_SSHD = os.path.join(CONFIG_DIR, 'filter.d', 'sshd.conf') + FILENAME_ZZZ_SSHD = os.path.join(TEST_FILES_DIR, 'zzz-sshd-obsolete-multiline.log') + FILTER_ZZZ_SSHD = os.path.join(TEST_CONFIG_DIR, 'filter.d', 'zzz-sshd-obsolete-multiline.conf') + + FILENAME_ZZZ_GEN = os.path.join(TEST_FILES_DIR, "logs", "zzz-generic-example") + FILTER_ZZZ_GEN = os.path.join(TEST_CONFIG_DIR, 'filter.d', 'zzz-generic-example.conf') def setUp(self): """Call before every test case.""" @@ -96,7 +106,7 @@ class Fail2banRegexTest(LogCaptureTestCase): (opts, args, fail2banRegex) = _Fail2banRegex( "test", r".** from <HOST>$" ) - self.assertFalse(fail2banRegex.start(opts, args)) + self.assertFalse(fail2banRegex.start(args)) self.assertLogged("Unable to compile regular expression") def testWrongIngnoreRE(self): @@ -104,7 +114,7 @@ class Fail2banRegexTest(LogCaptureTestCase): "--datepattern", "{^LN-BEG}EPOCH", "test", r".*? from <HOST>$", r".**" ) - self.assertFalse(fail2banRegex.start(opts, args)) + self.assertFalse(fail2banRegex.start(args)) self.assertLogged("Unable to compile regular expression") def testDirectFound(self): @@ -114,7 +124,7 @@ class Fail2banRegexTest(LogCaptureTestCase): "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0", r"Authentication failure for .*? from <HOST>$" ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 1 lines, 0 ignored, 1 matched, 0 missed') def testDirectNotFound(self): @@ -123,7 +133,7 @@ class Fail2banRegexTest(LogCaptureTestCase): "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0", r"XYZ from <HOST>$" ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 1 lines, 0 ignored, 0 matched, 1 missed') def testDirectIgnored(self): @@ -133,7 +143,7 @@ class Fail2banRegexTest(LogCaptureTestCase): r"Authentication failure for .*? from <HOST>$", r"kevin from 192.0.2.0$" ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 1 lines, 1 ignored, 0 matched, 0 missed') def testDirectRE_1(self): @@ -143,7 +153,7 @@ class Fail2banRegexTest(LogCaptureTestCase): Fail2banRegexTest.FILENAME_01, Fail2banRegexTest.RE_00 ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 19 lines, 0 ignored, 13 matched, 6 missed') self.assertLogged('Error decoding line'); @@ -159,7 +169,7 @@ class Fail2banRegexTest(LogCaptureTestCase): Fail2banRegexTest.FILENAME_01, Fail2banRegexTest.RE_00 ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 19 lines, 0 ignored, 16 matched, 3 missed') def testDirectRE_1raw_noDns(self): @@ -169,7 +179,7 @@ class Fail2banRegexTest(LogCaptureTestCase): Fail2banRegexTest.FILENAME_01, Fail2banRegexTest.RE_00 ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 19 lines, 0 ignored, 13 matched, 6 missed') def testDirectRE_2(self): @@ -179,28 +189,128 @@ class Fail2banRegexTest(LogCaptureTestCase): Fail2banRegexTest.FILENAME_02, Fail2banRegexTest.RE_00 ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 13 lines, 0 ignored, 5 matched, 8 missed') def testVerbose(self): (opts, args, fail2banRegex) = _Fail2banRegex( "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", + "--timezone", "UTC+0200", "--verbose", "--verbose-date", "--print-no-missed", Fail2banRegexTest.FILENAME_02, Fail2banRegexTest.RE_00 ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 13 lines, 0 ignored, 5 matched, 8 missed') self.assertLogged('141.3.81.106 Sun Aug 14 11:53:59 2005') self.assertLogged('141.3.81.106 Sun Aug 14 11:54:59 2005') + def testVerboseFullSshd(self): + (opts, args, fail2banRegex) = _Fail2banRegex( + "-l", "notice", # put down log-level, because of too many debug-messages + "-v", "--verbose-date", "--print-all-matched", + "-c", CONFIG_DIR, + Fail2banRegexTest.FILENAME_SSHD, "sshd" + ) + self.assertTrue(fail2banRegex.start(args)) + # test failure line and not-failure lines both presents: + self.assertLogged("[29116]: User root not allowed because account is locked", + "[29116]: Received disconnect from 1.2.3.4", all=True) + + def testFastSshd(self): + (opts, args, fail2banRegex) = _Fail2banRegex( + "-l", "notice", # put down log-level, because of too many debug-messages + "--print-all-matched", + "-c", CONFIG_DIR, + Fail2banRegexTest.FILENAME_ZZZ_SSHD, "sshd.conf[mode=normal]" + ) + self.assertTrue(fail2banRegex.start(args)) + # test failure line and all not-failure lines presents: + self.assertLogged( + "[29116]: Connection from 192.0.2.4", + "[29116]: User root not allowed because account is locked", + "[29116]: Received disconnect from 192.0.2.4", all=True) + + def testMultilineSshd(self): + # by the way test of missing lines by multiline in `for bufLine in orgLineBuffer[int(fullBuffer):]` + (opts, args, fail2banRegex) = _Fail2banRegex( + "-l", "notice", # put down log-level, because of too many debug-messages + "--print-all-matched", "--print-all-missed", + "-c", os.path.dirname(Fail2banRegexTest.FILTER_ZZZ_SSHD), + Fail2banRegexTest.FILENAME_ZZZ_SSHD, os.path.basename(Fail2banRegexTest.FILTER_ZZZ_SSHD) + ) + self.assertTrue(fail2banRegex.start(args)) + # test "failure" line presents (2nd part only, because multiline fewer precise): + self.assertLogged( + "[29116]: Received disconnect from 192.0.2.4", all=True) + + def testFullGeneric(self): + # by the way test of ignoreregex (specified in filter file)... + (opts, args, fail2banRegex) = _Fail2banRegex( + "-l", "notice", # put down log-level, because of too many debug-messages + Fail2banRegexTest.FILENAME_ZZZ_GEN, Fail2banRegexTest.FILTER_ZZZ_GEN+"[mode=test]" + ) + self.assertTrue(fail2banRegex.start(args)) + + def testDirectMultilineBuf(self): + # test it with some pre-lines also to cover correct buffer scrolling (all multi-lines printed): + for preLines in (0, 20): + self.pruneLog("[test-phase %s]" % preLines) + (opts, args, fail2banRegex) = _Fail2banRegex( + "--usedns", "no", "-d", "^Epoch", "--print-all-matched", "--maxlines", "5", + ("1490349000 TEST-NL\n"*preLines) + + "1490349000 FAIL\n1490349000 TEST1\n1490349001 TEST2\n1490349001 HOST 192.0.2.34", + r"^\s*FAIL\s*$<SKIPLINES>^\s*HOST <HOST>\s*$" + ) + self.assertTrue(fail2banRegex.start(args)) + self.assertLogged('Lines: %s lines, 0 ignored, 2 matched, %s missed' % (preLines+4, preLines+2)) + # both matched lines were printed: + self.assertLogged("| 1490349000 FAIL", "| 1490349001 HOST 192.0.2.34", all=True) + + + def testDirectMultilineBufDebuggex(self): + (opts, args, fail2banRegex) = _Fail2banRegex( + "--usedns", "no", "-d", "^Epoch", "--debuggex", "--print-all-matched", "--maxlines", "5", + "1490349000 FAIL\n1490349000 TEST1\n1490349001 TEST2\n1490349001 HOST 192.0.2.34", + r"^\s*FAIL\s*$<SKIPLINES>^\s*HOST <HOST>\s*$" + ) + self.assertTrue(fail2banRegex.start(args)) + self.assertLogged('Lines: 4 lines, 0 ignored, 2 matched, 2 missed') + # the sequence in args-dict is currently undefined (so can be 1st argument) + self.assertLogged("&flags=m", "?flags=m") + + def testSinglelineWithNLinContent(self): + # + (opts, args, fail2banRegex) = _Fail2banRegex( + "--usedns", "no", "-d", "^Epoch", "--print-all-matched", + "1490349000 FAIL: failure\nhost: 192.0.2.35", + r"^\s*FAIL:\s*.*\nhost:\s+<HOST>$" + ) + self.assertTrue(fail2banRegex.start(args)) + self.assertLogged('Lines: 1 lines, 0 ignored, 1 matched, 0 missed') + + + def testWrongFilterFile(self): + # use test log as filter file to cover eror cases... + (opts, args, fail2banRegex) = _Fail2banRegex( + Fail2banRegexTest.FILENAME_ZZZ_GEN, Fail2banRegexTest.FILENAME_ZZZ_GEN + ) + self.assertFalse(fail2banRegex.start(args)) + + def _reset(self): + # reset global warn-counter: + from ..server.filter import _decode_line_warn + _decode_line_warn.clear() + def testWronChar(self): + self._reset() (opts, args, fail2banRegex) = _Fail2banRegex( + "-l", "notice", # put down log-level, because of too many debug-messages "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", Fail2banRegexTest.FILENAME_WRONGCHAR, Fail2banRegexTest.FILTER_SSHD ) - self.assertTrue(fail2banRegex.start(opts, args)) + self.assertTrue(fail2banRegex.start(args)) self.assertLogged('Lines: 4 lines, 0 ignored, 2 matched, 2 missed') self.assertLogged('Error decoding line') @@ -210,13 +320,17 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('Nov 8 00:16:12 main sshd[32547]: pam_succeed_if(sshd:auth): error retrieving information about user llinco') def testWronCharDebuggex(self): + self._reset() (opts, args, fail2banRegex) = _Fail2banRegex( + "-l", "notice", # put down log-level, because of too many debug-messages "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--debuggex", "--print-all-matched", - Fail2banRegexTest.FILENAME_WRONGCHAR, Fail2banRegexTest.FILTER_SSHD + Fail2banRegexTest.FILENAME_WRONGCHAR, Fail2banRegexTest.FILTER_SSHD, + r"llinco[^\\]" ) - self.assertTrue(fail2banRegex.start(opts, args)) - self.assertLogged('Lines: 4 lines, 0 ignored, 2 matched, 2 missed') + self.assertTrue(fail2banRegex.start(args)) + self.assertLogged('Error decoding line') + self.assertLogged('Lines: 4 lines, 1 ignored, 2 matched, 1 missed') self.assertLogged('https://') diff --git a/fail2ban/tests/failmanagertestcase.py b/fail2ban/tests/failmanagertestcase.py index ed68e2a8..18f2c545 100644 --- a/fail2ban/tests/failmanagertestcase.py +++ b/fail2ban/tests/failmanagertestcase.py @@ -42,7 +42,8 @@ class AddFailure(unittest.TestCase): def tearDown(self): """Call after every test case.""" - + super(AddFailure, self).tearDown() + def _addDefItems(self): self.__items = [[u'193.168.0.128', 1167605999.0], [u'193.168.0.128', 1167605999.0], diff --git a/fail2ban/tests/files/logs/apache-auth b/fail2ban/tests/files/logs/apache-auth index 29de57eb..d430e291 100644 --- a/fail2ban/tests/files/logs/apache-auth +++ b/fail2ban/tests/files/logs/apache-auth @@ -124,3 +124,15 @@ # failJSON: { "time": "2013-11-18T22:39:33", "match": true , "host": "91.49.82.139" } [Mon Nov 18 22:39:33 2013] [error] [client 91.49.82.139] user gg not found: /, referer: http://sj.hopto.org/management.html + +# filterOptions: {"logging": "syslog"} + +# failJSON: { "time": "2005-02-15T16:23:00", "match": true , "host": "192.0.2.1", "desc": "using syslog (ErrorLog syslog)" } +Feb 15 16:23:00 srv httpd[22034]: [authz_core:error] [pid 22034] [client 192.0.2.1:58585] AH01630: client denied by server configuration: /home/www/ +# failJSON: { "time": "2005-02-15T16:23:40", "match": true , "host": "192.0.2.2", "desc": "using syslog (ErrorLog syslog)" } +Feb 15 16:23:40 srv httpd/backend1[22034]: [authz_core:error] [pid 22036] [client 192.0.2.2:59392] AH01630: client denied by server configuration: /home/backend1/ +# failJSON: { "time": "2005-02-15T16:54:53", "match": true , "host": "192.0.2.3", "desc": "using syslog (ErrorLog syslog)" } +Feb 15 16:54:53 tools apache2[18154]: [:error] [pid 18154:tid 140680873617152] [client 192.0.2.3:48154] AH01630: client denied by server configuration: /var/www + +# failJSON: { "time": "2005-02-16T22:32:48", "match": true , "host": "127.0.0.1" } +Feb 16 22:32:48 srv httpd[22034]: [error] [client 127.0.0.1] user wrongusername not found: /basic/file diff --git a/fail2ban/tests/files/logs/apache-modsecurity b/fail2ban/tests/files/logs/apache-modsecurity index d46d8ab4..3ca2e074 100644 --- a/fail2ban/tests/files/logs/apache-modsecurity +++ b/fail2ban/tests/files/logs/apache-modsecurity @@ -1,5 +1,5 @@ # failJSON: { "time": "2013-12-23T13:12:31", "match": true , "host": "173.255.225.101" } [Mon Dec 23 13:12:31 2013] [error] [client 173.255.225.101] ModSecurity: [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"][tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [hostname "www.mysite.net"] [uri "/"] [unique_id "Urf@f12qgHIAACrFOlgAAABA"] -# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69" } -[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] +# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69", "desc": "additional entry (and exact one space)" } +[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] diff --git a/fail2ban/tests/files/logs/asterisk b/fail2ban/tests/files/logs/asterisk index 13808592..5e846111 100644 --- a/fail2ban/tests/files/logs/asterisk +++ b/fail2ban/tests/files/logs/asterisk @@ -70,6 +70,9 @@ Nov 4 18:30:40 localhost asterisk[32229]: NOTICE[32257]: chan_sip.c:23417 in han # failJSON: { "time": "2016-01-28T10:34:33", "match": true , "host": "1.2.3.4" } [2016-01-28 10:34:33] NOTICE[3477][C-000003c3] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '' rejected because extension not found in context 'my-context'. +# failJSON: { "time": "2016-05-15T22:53:00", "match": true , "host": "192.0.2.4" } +[2016-05-15 22:53:00] SECURITY[26428] res_security_log.c: SecurityEvent="FailedACL",EventTV="2016-05-15T22:53:00.203+0300",Severity="Error",Service="AMI",EventVersion="1",AccountID="admin",SessionID="0x7fb580001518",LocalAddress="IPV4/TCP/0.0.0.0/5038",RemoteAddress="IPV4/TCP/192.0.2.4/62389",SessionTV="1970-01-01T03:00:00.000+0300" + # Failed authentication with pjsip on Asterisk 13+ # failJSON: { "time": "2016-05-23T10:18:16", "match": true , "host": "1.2.3.4" } [2016-05-23 10:18:16] NOTICE[19388] res_pjsip/pjsip_distributor.c: Request from '"1000" <sip:1000@10.0.0.1>' failed for '1.2.3.4:48336' (callid: 276666022) - No matching endpoint found @@ -84,3 +87,25 @@ Nov 4 18:30:40 localhost asterisk[32229]: NOTICE[32257]: chan_sip.c:23417 in han # Failed authentication with pjsip on Asterisk 13+ # failJSON: { "time": "2016-06-08T23:40:26", "match": true , "host": "2.3.4.5" } [2016-06-08 23:40:26] NOTICE[32497] res_pjsip/pjsip_distributor.c: Request from '"317" <sip:317@1.2.3.4>' failed for '2.3.4.5:5089' (callid: 206f178f-896564cb-57573f49@1.2.3.4) - No matching endpoint found + +# failJSON: { "time": "2016-06-09T00:01:02", "match": true , "host": "192.0.2.1" } +[2016-06-09 00:01:02] NOTICE [22382] manager.c: 192.0.2.1 failed to authenticate as 'admin' + +# Check AMI logs +# failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.4" } +[2016-05-06 07:08:09] NOTICE[31554] manager.c: 192.0.2.4 tried to authenticate with nonexistent user 'opennms' +# failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.5" } +[2016-05-06 07:08:09] NOTICE[6772] manager.c: 192.0.2.5 failed to authenticate as 'Admin' + +# PJSip Errors +# failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.6" } +[2016-05-06 07:08:09] NOTICE[17103] res_pjsip/pjsip_distributor.c: Request from '"test1" <sip:test1@2.3.4.5>' failed for '192.0.2.6:5678' (callid: deadbeef) - No matching endpoint found + +# # FreePBX Warnings +# #_dis_failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.4" } +# [2016-05-06 07:08:09] WARNING[6410][C-00000bac] Ext. 50048943556071: Friendly Scanner from 192.0.2.4 +# #_dis_failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.5" } +# [2016-05-06 07:08:09] WARNING[6410][C-00000bac] Ext. s: Friendly Scanner from 192.0.2.5 +# #_dis_failJSON: { "time": "2016-05-06T07:08:09", "match": true, "host": "192.0.2.6" } +# [2016-05-06 07:08:09] WARNING[6410][C-00000bac] Ext. +012345: Friendly Scanner from 192.0.2.6 +# # Yes, this does have quotes around it. diff --git a/fail2ban/tests/files/logs/courier-auth b/fail2ban/tests/files/logs/courier-auth index e3d0d8c3..3505e109 100644 --- a/fail2ban/tests/files/logs/courier-auth +++ b/fail2ban/tests/files/logs/courier-auth @@ -6,3 +6,5 @@ Apr 23 21:59:38 dns2 pop3d: LOGIN FAILED, user=info@example.com, ip=[::ffff:198. Nov 13 08:11:53 server imapd-ssl: LOGIN FAILED, user=user@domain.tld, ip=[::ffff:198.51.100.33] # failJSON: { "time": "2005-04-17T19:17:11", "match": true , "host": "1.2.3.4" } Apr 17 19:17:11 SERVER courierpop3login: LOGIN FAILED, user=USER@EXAMPLE.org, ip=[::ffff:1.2.3.4] +# failJSON: { "time": "2005-04-17T19:17:12", "match": true , "host": "192.0.2.4" } +Apr 17 19:17:12 server imapd-ssl: LOGIN FAILED, method=PLAIN, ip=[::ffff:192.0.2.4] diff --git a/fail2ban/tests/files/logs/cyrus-imap b/fail2ban/tests/files/logs/cyrus-imap index f1edff06..c0ec54cb 100644 --- a/fail2ban/tests/files/logs/cyrus-imap +++ b/fail2ban/tests/files/logs/cyrus-imap @@ -16,3 +16,6 @@ Dec 30 16:03:27 somehost imapd[2517]: badlogin: local-somehost[1.2.3.4] OTP [SAS Jul 17 22:55:56 derry cyrus/imaps[7568]: badlogin: serafinat.xxxxxx [1.2.3.4] plain [SASL(-13): user not found: user: pressy@derry property: cmusaslsecretPLAIN not found in sasldb] # failJSON: { "time": "2005-07-18T16:46:42", "match": true , "host": "1.2.3.4" } Jul 18 16:46:42 derry cyrus/imaps[27449]: badlogin: serafinat.xxxxxx [1.2.3.4] PLAIN [SASL(-13): user not found: Password verification failed] + +# failJSON: { "time": "2005-03-08T05:25:21", "match": true , "host": "192.0.2.4", "desc": "entry without loginname/hostname before IP" } +Mar 8 05:25:21 host imap[22130]: badlogin: [192.0.2.4] plain [SASL(-13): authentication failure: Password verification failed]
\ No newline at end of file diff --git a/fail2ban/tests/files/logs/domino-smtp b/fail2ban/tests/files/logs/domino-smtp new file mode 100644 index 00000000..4987e7ea --- /dev/null +++ b/fail2ban/tests/files/logs/domino-smtp @@ -0,0 +1,8 @@ +# failJSON: { "time": "2005-07-03T23:07:20", "match": true , "host": "1.2.3.4" } +03-07-2005 23:07:20 SMTP Server: Authentication failed for user postmaster ; connecting host 1.2.3.4 +# failJSON: { "time": "2014-06-22T09:56:12", "match": true , "host": "1.2.3.4" } +[28325:00010-3735542592] 22-06-2014 09:56:12 smtp: postmaster [1.2.3.4] authentication failure using internet password +# failJSON: { "time": "2014-09-08T06:14:27", "match": true , "host": "1.2.3.4" } +08-09-2014 06:14:27 smtp: postmaster [1.2.3.4] authentication failure using internet password +# failJSON: { "time": "2016-11-07T22:21:20", "match": true , "host": "1.2.3.4" } +2016-11-07 22:21:20 smtp: postmaster [1.2.3.4] authentication failure using internet password diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 627b8dc8..1614ff8c 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -40,6 +40,9 @@ Jan 29 05:13:40 mail dovecot: auth-worker(31326): pam(username,1.2.3.4): unknown # failJSON: { "time": "2005-01-29T05:13:50", "match": true , "host": "1.2.3.4" } Jan 29 05:13:50 mail dovecot: auth: passwd-file(username,1.2.3.4): unknown user +# failJSON: { "time": "2005-01-29T13:54:06", "match": true , "host": "192.0.2.5" } +Jan 29 13:54:06 auth-worker(22401): Info: sql(admin@example.de,192.0.2.5,<n4JLdHNVngZGpV2j>): unknown user + # failJSON: { "time": "2005-04-19T05:22:20", "match": true , "host": "80.255.3.104" } Apr 19 05:22:20 vm5 auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=informix rhost=80.255.3.104 @@ -73,3 +76,8 @@ Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempt # failJSON: { "time": "2005-03-23T06:10:52", "match": true , "host": "52.37.139.121" } Mar 23 06:10:52 auth: Info: ldap(dog,52.37.139.121,): invalid credentials + +# failJSON: { "time": "2005-07-26T11:11:21", "match": true , "host": "192.0.2.1" } +Jul 26 11:11:21 hostname dovecot: imap-login: Disconnected: Too many invalid commands (tried to use disallowed plaintext auth): user=<test>, rip=192.0.2.1, lip=192.168.1.1, session=<S5dIdTFCDKUWWMbU> +# failJSON: { "time": "2005-07-26T11:12:19", "match": true , "host": "192.0.2.2" } +Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid commands (auth failed, 1 attempts in 17 secs): user=<test>, method=PLAIN, rip=192.0.2.2, lip=192.168.1.1, TLS, session=<g3ZKeDECFqlWWMbU> diff --git a/fail2ban/tests/files/logs/ejabberd-auth b/fail2ban/tests/files/logs/ejabberd-auth index e8e89fb8..9f03e6a4 100644 --- a/fail2ban/tests/files/logs/ejabberd-auth +++ b/fail2ban/tests/files/logs/ejabberd-auth @@ -9,3 +9,12 @@ I(<0.370.0>:ejabberd_listener:281) : (#Port<0.6910>) Accepted connection {{192,0 I(<0.1440.0>:ejabberd_c2s:813) : ({socket_state,tls,{tlssock,#Port<0.6910>,#Port<0.6912>},<0.1439.0>}) Failed authentication for user@example.com from IP 192.0.2.4 ({{192,0,2,4},12716}) # failJSON: { "time": "2014-01-07T18:09:08", "match": true , "host": "1.2.3.4" } 2014-01-07 18:09:08.512 [info] <0.22741.1>@ejabberd_c2s:wait_for_feature_request:662 ({socket_state,p1_tls,{tlssock,#Port<0.24718>,#Port<0.24720>},<0.22740.1>}) Failed authentication for test@example.com from IP 1.2.3.4 + +# new format: + +# failJSON: { "time": "2015-03-19T13:57:35", "match": true , "host": "192.0.2.6" } +2015-03-19 13:57:35.805 [info] <0.585.0>@ejabberd_c2s:wait_for_sasl_response:965 ({socket_state,p1_tls,{tlssock,#Port<0.6434>,#Port<0.6436>},<0.584.0>}) Failed authentication for robin@example.com from 192.0.2.6 + +# 17.06 "new" format: +# failJSON: { "time": "2017-07-29T08:24:04", "match": true , "host": "192.0.2.3" } +2017-07-29 08:24:04.773 [info] <0.6668.0>@ejabberd_c2s:handle_auth_failure:433 (http_bind|ejabberd_bosh) Failed c2s PLAIN authentication for test@example.ch from ::FFFF:192.0.2.3: Invalid username or password diff --git a/fail2ban/tests/files/logs/exim b/fail2ban/tests/files/logs/exim index 9053bf8d..8f3ac1bb 100644 --- a/fail2ban/tests/files/logs/exim +++ b/fail2ban/tests/files/logs/exim @@ -60,10 +60,19 @@ 2016-03-21 04:07:49 [25874] 1ahr79-0006jK-G9 SMTP connection from (voyeur.webair.com) [174.137.147.204]:44884 I=[172.89.0.6]:25 closed by DROP in ACL # failJSON: { "time": "2016-03-21T04:33:13", "match": true , "host": "206.214.71.53" } 2016-03-21 04:33:13 [26074] 1ahrVl-0006mY-79 SMTP connection from riveruse.com [206.214.71.53]:39865 I=[172.89.0.6]:25 closed by DROP in ACL +# failJSON: { "time": "2016-03-21T04:33:14", "match": true , "host": "192.0.2.33", "desc": "short form without optional session-id" } +2016-03-21 04:33:14 SMTP connection from (some.domain) [192.0.2.33] closed by DROP in ACL # failJSON: { "time": "2016-04-01T11:08:39", "match": true , "host": "192.0.2.1" } 2016-04-01 11:08:39 [18643] no MAIL in SMTP connection from host.example.com (SERVER) [192.0.2.1]:1418 I=[172.89.0.6]:25 D=34s C=EHLO,AUTH +# failJSON: { "time": "2016-04-01T11:08:40", "match": true , "host": "192.0.2.2" } +2016-04-01 11:08:40 [18643] no MAIL in SMTP connection from host.example.com (SERVER) [192.0.2.2]:1418 I=[172.89.0.6]:25 D=2m42s C=QUIT # failJSON: { "time": "2016-04-01T11:09:21", "match": true , "host": "192.0.2.1" } 2016-04-01 11:09:21 [18648] SMTP protocol error in "AUTH LOGIN" H=host.example.com (SERVER) [192.0.2.1]:4692 I=[172.89.0.6]:25 AUTH command used when not advertised # failJSON: { "time": "2016-03-27T16:48:48", "match": true , "host": "192.0.2.1" } 2016-03-27 16:48:48 [21478] 1akDqs-0005aQ-9b SMTP connection from host.example.com (SERVER) [192.0.2.1]:47714 I=[172.89.0.6]:25 closed by DROP in ACL + +# failJSON: { "time": "2017-04-23T22:45:59", "match": true , "host": "192.0.2.2", "desc": "optional part (...)" } +2017-04-23 22:45:59 fixed_login authenticator failed for bad.host.example.com [192.0.2.2]:54412 I=[172.89.0.6]:587: 535 Incorrect authentication data (set_id=user@example.com) +# failJSON: { "time": "2017-05-01T07:42:42", "match": true , "host": "192.0.2.3", "desc": "rejected RCPT - Unrouteable address" } +2017-05-01 07:42:42 H=some.rev.dns.if.found (the.connector.reports.this.name) [192.0.2.3] F=<some.name@some.domain> rejected RCPT <some.invalid.name@a.domain>: Unrouteable address diff --git a/fail2ban/tests/files/logs/haproxy-http-auth b/fail2ban/tests/files/logs/haproxy-http-auth index 298f1972..403a8083 100644 --- a/fail2ban/tests/files/logs/haproxy-http-auth +++ b/fail2ban/tests/files/logs/haproxy-http-auth @@ -2,3 +2,7 @@ Nov 14 22:45:27 test haproxy[760]: 192.168.33.1:58444 [14/Nov/2015:22:45:25.439] main app/app1 1939/0/1/0/1940 403 5168 - - ---- 3/3/0/0/0 0/0 "GET / HTTP/1.1" # failJSON: { "time": "2004-11-14T22:45:11", "match": true , "host": "192.168.33.1" } Nov 14 22:45:11 test haproxy[760]: 192.168.33.1:58430 [14/Nov/2015:22:45:11.608] main main/<NOSRV> -1/-1/-1/-1/0 401 248 - - PR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1" +# failJSON: { "time": "2004-11-14T22:45:11", "match": true , "host": "2001:db8::1234" } +Nov 14 22:45:11 test haproxy[760]: 2001:db8::1234:58430 [14/Nov/2015:22:45:11.608] main main/<NOSRV> -1/-1/-1/-1/0 401 248 - - PR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1" +# failJSON: { "time": "2004-11-14T22:45:11", "match": true , "host": "192.168.33.1" } +Nov 14 22:45:11 test haproxy[760]: ::ffff:192.168.33.1:58430 [14/Nov/2015:22:45:11.608] main main/<NOSRV> -1/-1/-1/-1/0 401 248 - - PR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1" diff --git a/fail2ban/tests/files/logs/kerio b/fail2ban/tests/files/logs/kerio index c9368c22..f7aa0eb7 100644 --- a/fail2ban/tests/files/logs/kerio +++ b/fail2ban/tests/files/logs/kerio @@ -25,5 +25,20 @@ # failJSON: { "time": "2013-12-13T01:11:04", "match": true, "host": "218.85.253.185" } [13/Dec/2013 01:11:04] Attempt to deliver to unknown recipient <marge@aplawrence.com>, from <yu@rrd.com>, IP address 218.85.253.185 +# failJSON: { "time": "2017-05-29T17:29:29", "match": true, "host": "185.140.108.56" } +[29/May/2017 17:29:29] IP address 185.140.108.56 found in DNS blacklist SpamCop, mail from <noreply-tjgqNffcgPfpbZtpDzasm@oakspaversusa.com> to <info@verinion.com> rejected +# failJSON: { "time": "2017-05-17T19:43:42", "match": true, "host": "185.140.108.26" } +[17/May/2017 19:43:42] SMTP: User printer@verinion.com doesn't exist. Attempt from IP address 185.140.108.26. +# failJSON: { "time": "2017-05-17T19:44:25", "match": true, "host": "184.171.168.211" } +[17/May/2017 19:44:25] Client with IP address 184.171.168.211 has no reverse DNS entry, connection rejected before SMTP greeting + +# failJSON: { "time": "2017-05-17T19:45:27", "match": true, "host": "170.178.167.136" } +[17/May/2017 19:45:27] Administration login into Web Administration from 170.178.167.136 failed: IP address not allowed + +# failJSON: { "time": "2017-05-17T22:14:57", "match": true, "host": "67.211.219.82" } +[17/May/2017 22:14:57] Message from IP address 67.211.219.82, sender <promo123@goodresponse.site> rejected: sender domain does not exist + +# failJSON: { "time": "2017-05-18T07:25:15", "match": true, "host": "212.92.127.112" } +[18/May/2017 07:25:15] Failed SMTP login from 212.92.127.112 with SASL method CRAM-MD5. diff --git a/fail2ban/tests/files/logs/mongodb-auth b/fail2ban/tests/files/logs/mongodb-auth new file mode 100644 index 00000000..8a308892 --- /dev/null +++ b/fail2ban/tests/files/logs/mongodb-auth @@ -0,0 +1,30 @@ +# failJSON: { "match": false } +2016-11-20T00:04:00.110+0100 [conn1] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin +# failJSON: { "time": "2016-11-20T00:04:00", "match": true , "host": "192.0.2.35" } +2016-11-20T00:04:00.111+0100 [conn1] end connection 192.0.2.35:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn5] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.171" } +2016-11-20T00:24:00.111+0100 [conn5] end connection 192.0.2.171:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn334] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.176" } +2016-11-20T00:24:00.111+0100 [conn334] end connection 192.0.2.176:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn56] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.1" } +2016-11-20T00:24:00.111+0100 [conn56] end connection 192.0.2.1:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T12:54:02.370+0100 [initandlisten] connection accepted from 127.0.0.1:58774 #2261 (1 connection now open) +# failJSON: { "match": false } +2016-11-20T12:54:02.370+0100 [conn2261] end connection 127.0.0.1:58774 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T13:07:49.781+0100 [conn2271] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# failJSON: { "time": "2016-11-20T13:07:49", "match": false , "host": "192.0.2.178" } +2016-11-20T13:07:49.834+0100 [conn2271] end connection 192.0.2.178:60268 (3 connections now open) + diff --git a/fail2ban/tests/files/logs/pam-generic b/fail2ban/tests/files/logs/pam-generic index e562ac7f..1740f0c8 100644 --- a/fail2ban/tests/files/logs/pam-generic +++ b/fail2ban/tests/files/logs/pam-generic @@ -1,17 +1,23 @@ -# failJSON: { "time": "2005-02-07T15:10:42", "match": true , "host": "192.168.1.1" } +# failJSON: { "time": "2005-02-07T15:10:42", "match": true , "host": "192.168.1.1", "user": "sample-user" } Feb 7 15:10:42 example pure-ftpd: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=pure-ftpd ruser=sample-user rhost=192.168.1.1 -# failJSON: { "time": "2005-05-12T09:47:54", "match": true , "host": "71-13-115-12.static.mdsn.wi.charter.com" } +# failJSON: { "time": "2005-05-12T09:47:54", "match": true , "host": "71-13-115-12.static.mdsn.wi.charter.com", "user": "root" } May 12 09:47:54 vaio sshd[16004]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=71-13-115-12.static.mdsn.wi.charter.com user=root # failJSON: { "time": "2005-05-12T09:48:03", "match": true , "host": "71-13-115-12.static.mdsn.wi.charter.com" } May 12 09:48:03 vaio sshd[16021]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=71-13-115-12.static.mdsn.wi.charter.com -# failJSON: { "time": "2005-05-15T18:02:12", "match": true , "host": "66.232.129.62" } +# failJSON: { "time": "2005-05-15T18:02:12", "match": true , "host": "66.232.129.62", "user": "mark" } May 15 18:02:12 localhost proftpd: (pam_unix) authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=66.232.129.62 user=mark # linux-pam messages before commit f0f9c4479303b5a9c37667cf07f58426dc081676 (release 0.99.2.0 ) - nolonger supported # failJSON: { "time": "2004-11-25T17:12:13", "match": false } Nov 25 17:12:13 webmail pop(pam_unix)[4920]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=192.168.10.3 user=mailuser -# failJSON: { "time": "2005-07-19T18:11:26", "match": true , "host": "www.google.com" } +# failJSON: { "time": "2005-07-19T18:11:26", "match": true , "host": "www.google.com", "user": "an8767" } Jul 19 18:11:26 srv2 vsftpd: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=an8767 rhost=www.google.com # failJSON: { "time": "2005-07-19T18:11:26", "match": true , "host": "www.google.com" } Jul 19 18:11:26 srv2 vsftpd: pam_unix: authentication failure; logname= uid=0 euid=0 tty=ftp ruser=an8767 rhost=www.google.com + + +# failJSON: { "time": "2005-07-19T18:11:50", "match": true , "host": "192.0.2.1", "user": "test rhost=192.0.2.151", "desc": "Injecting on username"} +Jul 19 18:11:50 srv2 daemon: pam_unix(auth): authentication failure; logname= uid=0 euid=0 tty=xxx ruser=test rhost=192.0.2.151 rhost=192.0.2.1 +# failJSON: { "time": "2005-07-19T18:11:52", "match": true , "host": "192.0.2.2", "user": "test rhost=192.0.2.152", "desc": "Injecting on username after host"} +Jul 19 18:11:52 srv2 daemon: pam_unix(auth): authentication failure; logname= uid=0 euid=0 tty=xxx ruser= rhost=192.0.2.2 user=test rhost=192.0.2.152 diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index 78f72287..54b8be99 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -1,3 +1,5 @@ +# filterOptions: [{}, {"mode": "normal"}, {"mode": "aggressive"}] + # per https://github.com/fail2ban/fail2ban/issues/125 # and https://github.com/fail2ban/fail2ban/issues/126 # failJSON: { "time": "2005-02-21T09:21:54", "match": true , "host": "192.0.43.10" } @@ -35,3 +37,102 @@ Jan 31 13:55:24 xxx postfix-incoming/smtpd[3462]: NOQUEUE: reject: EHLO from s27 # failJSON: { "time": "2005-04-12T02:24:11", "match": true , "host": "62.138.2.143" } Apr 12 02:24:11 xxx postfix/smtps/smtpd[42]: NOQUEUE: reject: EHLO from astra4139.startdedicated.de[62.138.2.143]: 504 5.5.2 <User>: Helo command rejected: need fully-qualified hostname; proto=SMTP helo=<User> + +# failJSON: { "time": "2005-06-12T08:58:35", "match": true , "host": "1.2.3.4" } +Jun 12 08:58:35 xxx postfix/smtpd[27296]: NOQUEUE: reject: RCPT from unknown[1.2.3.4]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [2.3.4.5]; from=<meow@kitty.com> to=<kitty@meow.com> proto=ESMTP helo=<kitty.com> + +# failJSON: { "time": "2005-06-12T08:58:35", "match": true , "host": "1.2.3.4" } +Jun 12 08:58:35 xxx postfix/smtpd[2931]: NOQUEUE: reject: RCPT from unknown[1.2.3.4]: 450 4.7.1 <kitty.com>: Helo command rejected: Host not found; from=<meow@kitty.com> to=<kitty@meow.com> proto=SMTP helo=<kitty.com> + +# failJSON: { "time": "2005-06-12T08:58:35", "match": true , "host": "1.2.3.4" } +Jun 12 08:58:35 xxx postfix/smtpd[13533]: improper command pipelining after AUTH from unknown[1.2.3.4]: QUIT + +# --------------------------------------- +# Test-cases of postfix-postscreen: +# --------------------------------------- + +# failJSON: { "time": "2005-05-05T15:51:11", "match": true , "host": "216.245.194.173", "desc": "postfix postscreen / gh-1764" } +May 5 15:51:11 xxx postfix/postscreen[1148]: NOQUEUE: reject: RCPT from [216.245.194.173]:60591: 550 5.7.1 Service unavailable; client [216.245.194.173] blocked using rbl.example.com; from=<spammer@example.com>, to=<goodguy@example.com>, proto=ESMTP, helo=<badguy.example.com> + +# --------------------------------------- +# Test-cases of postfix-rbl: +# --------------------------------------- +# filterOptions: [{}, {"mode": "rbl"}, {"mode": "aggressive"}] + +# failJSON: { "time": "2004-12-30T18:19:15", "match": true , "host": "93.184.216.34" } +Dec 30 18:19:15 xxx postfix/smtpd[1574]: NOQUEUE: reject: RCPT from badguy.example.com[93.184.216.34]: 454 4.7.1 Service unavailable; Client host [93.184.216.34] blocked using rbl.example.com; http://www.example.com/query?ip=93.184.216.34; from=<spammer@example.com> to=<goodguy@example.com> proto=ESMTP helo=<badguy.example.com> + +# failJSON: { "time": "2004-12-30T18:19:15", "match": true , "host": "93.184.216.34" } +Dec 30 18:19:15 xxx postfix-incoming/smtpd[1574]: NOQUEUE: reject: RCPT from badguy.example.com[93.184.216.34]: 454 4.7.1 Service unavailable; Client host [93.184.216.34] blocked using rbl.example.com; http://www.example.com/query?ip=93.184.216.34; from=<spammer@example.com> to=<goodguy@example.com> proto=ESMTP helo=<badguy.example.com> + +# failJSON: { "time": "2005-02-07T12:25:45", "match": true , "host": "87.236.233.182" } +Feb 7 12:25:45 xxx12345 postfix/smtpd[13275]: NOQUEUE: reject: RCPT from unknown[87.236.233.182]: 554 5.7.1 Service unavailable; Client host [87.236.233.182] blocked using rbl.example.com; https://www.example.com/query/ip/87.236.233.182; from=<spammer@example.com> to=<goodguy@example.com> proto=SMTP helo=<WIN-5N8GBBS0R5I> + +# --------------------------------------- +# Test-cases of postfix-sasl: +# --------------------------------------- +# filterOptions: [{"mode": "auth"}, {"mode": "aggressive"}] + +#1 Example from postfix from dbts #507990 +# failJSON: { "time": "2004-12-02T22:24:22", "match": true , "host": "114.44.142.233" } +Dec 2 22:24:22 hel postfix/smtpd[7676]: warning: 114-44-142-233.dynamic.hinet.net[114.44.142.233]: SASL CRAM-MD5 authentication failed: PDc3OTEwNTkyNTEyMzA2NDIuMTIyODI1MzA2MUBoZWw+ +#2 Example from postfix from dbts #573314 +# failJSON: { "time": "2005-03-10T13:33:30", "match": true , "host": "1.1.1.1" } +Mar 10 13:33:30 gandalf postfix/smtpd[3937]: warning: HOSTNAME[1.1.1.1]: SASL LOGIN authentication failed: authentication failure + +#3 Example from postfix post-debian changes to rename to add "submission" to syslog name +# failJSON: { "time": "2004-09-06T00:44:56", "match": true , "host": "82.221.106.233" } +Sep 6 00:44:56 trianon postfix/submission/smtpd[11538]: warning: unknown[82.221.106.233]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 + +#4 Example from postfix post-debian changes to rename to add "submission" to syslog name + downcase +# failJSON: { "time": "2004-09-06T00:44:57", "match": true , "host": "82.221.106.233" } +Sep 6 00:44:57 trianon postfix/submission/smtpd[11538]: warning: unknown[82.221.106.233]: SASL login authentication failed: UGFzc3dvcmQ6 + +#5 Example to add : +# failJSON: { "time": "2005-01-29T08:11:45", "match": true , "host": "1.1.1.1" } +Jan 29 08:11:45 mail postfix/smtpd[10752]: warning: unknown[1.1.1.1]: SASL LOGIN authentication failed: Password: + +# failJSON: { "time": "2005-01-29T08:11:45", "match": true , "host": "1.1.1.1" } +Jan 29 08:11:45 mail postfix-incoming/smtpd[10752]: warning: unknown[1.1.1.1]: SASL LOGIN authentication failed: Password: + +# failJSON: { "time": "2005-04-12T02:24:11", "match": true , "host": "62.138.2.143" } +Apr 12 02:24:11 xxx postfix/smtps/smtpd[42]: warning: astra4139.startdedicated.de[62.138.2.143]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 + +# failJSON: { "time": "2005-08-03T15:30:49", "match": true , "host": "98.191.84.74" } +Aug 3 15:30:49 ksusha postfix/smtpd[17041]: warning: mail.foldsandwalker.com[98.191.84.74]: SASL Plain authentication failed: + +# failJSON: { "time": "2004-11-04T09:11:01", "match": true , "host": "192.0.2.150", "desc": "without reason for fail, see gh-1245" } +Nov 4 09:11:01 mail postfix/submission/smtpd[27133]: warning: unknown[192.0.2.150]: SASL PLAIN authentication failed: + +#6 Example to ignore because due to a failed attempt to connect to authentication service - no malicious activities whatsoever +# failJSON: { "match": false } +Feb 3 08:29:28 mail postfix/smtpd[21022]: warning: unknown[1.1.1.1]: SASL LOGIN authentication failed: Connection lost to authentication server + +# filterOptions: [{"mode": "auth"}] + +# failJSON: { "match": false, "desc": "not aggressive" } +Jan 14 16:18:16 xxx postfix/smtpd[14933]: warning: host[192.0.2.5]: SASL CRAM-MD5 authentication failed: Invalid authentication mechanism + +# filterOptions: [{"mode": "aggressive"}] + +# failJSON: { "time": "2005-01-14T16:18:16", "match": true , "host": "192.0.2.5", "desc": "aggressive only" } +Jan 14 16:18:16 xxx postfix/smtpd[14933]: warning: host[192.0.2.5]: SASL CRAM-MD5 authentication failed: Invalid authentication mechanism + +# --------------------------------------- +# Test-cases of postfix DDOS mode: +# --------------------------------------- + +# filterOptions: [{"mode": "ddos"}, {"mode": "aggressive"}] + +# failJSON: { "time": "2005-02-18T09:45:10", "match": true , "host": "192.0.2.10" } +Feb 18 09:45:10 xxx postfix/smtpd[42]: lost connection after CONNECT from spammer.example.com[192.0.2.10] +# failJSON: { "time": "2005-02-18T09:45:12", "match": true , "host": "192.0.2.42" } +Feb 18 09:45:12 xxx postfix/smtpd[42]: lost connection after STARTTLS from spammer.example.com[192.0.2.42] +# failJSON: { "time": "2005-02-18T09:48:04", "match": true , "host": "192.0.2.23" } +Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[192.0.2.23] +# failJSON: { "time": "2005-02-18T09:48:04", "match": true , "host": "192.0.2.23" } +Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[192.0.2.23] + +# filterOptions: [{}, {"mode": "ddos"}, {"mode": "aggressive"}] +# failJSON: { "match": false, "desc": "don't affect lawful data (sporadical connection aborts within DATA-phase, see gh-1813 for discussion)" } +Feb 18 09:50:05 xxx postfix/smtpd[42]: lost connection after DATA from good-host.example.com[192.0.2.10] diff --git a/fail2ban/tests/files/logs/postfix-rbl b/fail2ban/tests/files/logs/postfix-rbl deleted file mode 100644 index eff01bf9..00000000 --- a/fail2ban/tests/files/logs/postfix-rbl +++ /dev/null @@ -1,5 +0,0 @@ -# failJSON: { "time": "2004-12-30T18:19:15", "match": true , "host": "93.184.216.34" } -Dec 30 18:19:15 xxx postfix/smtpd[1574]: NOQUEUE: reject: RCPT from badguy.example.com[93.184.216.34]: 454 4.7.1 Service unavailable; Client host [93.184.216.34] blocked using rbl.example.com; http://www.example.com/query?ip=93.184.216.34; from=<spammer@example.com> to=<goodguy@example.com> proto=ESMTP helo=<badguy.example.com> - -# failJSON: { "time": "2004-12-30T18:19:15", "match": true , "host": "93.184.216.34" } -Dec 30 18:19:15 xxx postfix-incoming/smtpd[1574]: NOQUEUE: reject: RCPT from badguy.example.com[93.184.216.34]: 454 4.7.1 Service unavailable; Client host [93.184.216.34] blocked using rbl.example.com; http://www.example.com/query?ip=93.184.216.34; from=<spammer@example.com> to=<goodguy@example.com> proto=ESMTP helo=<badguy.example.com> diff --git a/fail2ban/tests/files/logs/postfix-sasl b/fail2ban/tests/files/logs/postfix-sasl deleted file mode 100644 index cdcb5121..00000000 --- a/fail2ban/tests/files/logs/postfix-sasl +++ /dev/null @@ -1,32 +0,0 @@ -#1 Example from postfix from dbts #507990 -# failJSON: { "time": "2004-12-02T22:24:22", "match": true , "host": "114.44.142.233" } -Dec 2 22:24:22 hel postfix/smtpd[7676]: warning: 114-44-142-233.dynamic.hinet.net[114.44.142.233]: SASL CRAM-MD5 authentication failed: PDc3OTEwNTkyNTEyMzA2NDIuMTIyODI1MzA2MUBoZWw+ -#2 Example from postfix from dbts #573314 -# failJSON: { "time": "2005-03-10T13:33:30", "match": true , "host": "1.1.1.1" } -Mar 10 13:33:30 gandalf postfix/smtpd[3937]: warning: HOSTNAME[1.1.1.1]: SASL LOGIN authentication failed: authentication failure - -#3 Example from postfix post-debian changes to rename to add "submission" to syslog name -# failJSON: { "time": "2004-09-06T00:44:56", "match": true , "host": "82.221.106.233" } -Sep 6 00:44:56 trianon postfix/submission/smtpd[11538]: warning: unknown[82.221.106.233]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 - -#4 Example from postfix post-debian changes to rename to add "submission" to syslog name + downcase -# failJSON: { "time": "2004-09-06T00:44:57", "match": true , "host": "82.221.106.233" } -Sep 6 00:44:57 trianon postfix/submission/smtpd[11538]: warning: unknown[82.221.106.233]: SASL login authentication failed: UGFzc3dvcmQ6 - -#5 Example to add : -# failJSON: { "time": "2005-01-29T08:11:45", "match": true , "host": "1.1.1.1" } -Jan 29 08:11:45 mail postfix/smtpd[10752]: warning: unknown[1.1.1.1]: SASL LOGIN authentication failed: Password: - -#6 Example to ignore because due to a failed attempt to connect to authentication service - no malicious activities whatsoever -# failJSON: { "time": "2005-02-03T08:29:28", "match": false , "host": "1.1.1.1" } -Feb 3 08:29:28 mail postfix/smtpd[21022]: warning: unknown[1.1.1.1]: SASL LOGIN authentication failed: Connection lost to authentication server - -# failJSON: { "time": "2005-01-29T08:11:45", "match": true , "host": "1.1.1.1" } -Jan 29 08:11:45 mail postfix-incoming/smtpd[10752]: warning: unknown[1.1.1.1]: SASL LOGIN authentication failed: Password: - -# failJSON: { "time": "2005-04-12T02:24:11", "match": true , "host": "62.138.2.143" } -Apr 12 02:24:11 xxx postfix/smtps/smtpd[42]: warning: astra4139.startdedicated.de[62.138.2.143]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 - -# failJSON: { "time": "2005-08-03T15:30:49", "match": true , "host": "98.191.84.74" } -Aug 3 15:30:49 ksusha postfix/smtpd[17041]: warning: mail.foldsandwalker.com[98.191.84.74]: SASL Plain authentication failed: - diff --git a/fail2ban/tests/files/logs/roundcube-auth b/fail2ban/tests/files/logs/roundcube-auth index 26868c3e..f3f762d2 100644 --- a/fail2ban/tests/files/logs/roundcube-auth +++ b/fail2ban/tests/files/logs/roundcube-auth @@ -8,19 +8,27 @@ Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 1. # Made up to attempts to inject a DoS on the server. Assume the user can manipulate the IMAP error response # # user = admin from 127.0.0.1 -# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4" } +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4", "desc": "Injecting on username 1" } Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 127.0.0.1 from 1.2.3.4. AUTHENTICATE PLAIN: A0002 NO Login failed. in /usr/share/roundcube/program/include/rcube_imap.php on line 205 (POST /wmail/?_task=login&_action=login) +# user = admin from 127.0.0.1. +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4", "desc": "Injecting on username 1 (with dot)" } +Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 127.0.0.1. from 1.2.3.4. AUTHENTICATE PLAIN: A0002 NO Login failed. in /usr/share/roundcube/program/include/rcube_imap.php on line 205 (POST /wmail/?_task=login&_action=login) +# # # IMAP server logs user=${username} -# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4" } +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4", "desc": "Injecting on username 2" } Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 127.0.0.1 from 1.2.3.4. AUTHENTICATE PLAIN: A0002 NO Login failed. user=admin from 127.0.0.1 in /usr/share/roundcube/program/include/rcube_imap.php on line 205 (POST /wmail/?_task=login&_action=login) # +# IMAP server logs user=${username} +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4", "desc": "Injecting on username 2 (with dot)" } +Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 127.0.0.1. from 1.2.3.4. AUTHENTICATE PLAIN: A0002 NO Login failed. user=admin from 127.0.0.1. in /usr/share/roundcube/program/include/rcube_imap.php on line 205 (POST /wmail/?_task=login&_action=login) +# # Old roundcube version - no IMAP response -# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4" } +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4", "desc": "Injecting on username 3" } Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 127.0.0.1 from 1.2.3.4 # # user = admin from 127.0.0.1 in -# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4" } +# failJSON: { "time": "2005-07-11T03:06:37", "match": true , "host": "1.2.3.4", "desc": "Injecting on username 4" } Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 127.0.0.1 in from 1.2.3.4. AUTHENTICATE PLAIN: A0002 NO Login failed. user=admin from 127.0.0.1 in in /usr/share/roundcube/program/include/rcube_imap.php on line 205 (POST /wmail/?_task=login&_action=login) # Roundcube 1.0.5 CentOS 6 (/var/log/roundcubemail/errors) @@ -40,3 +48,9 @@ Jul 11 03:06:37 myhostname roundcube: IMAP Error: Login failed for admin from 12 # Roundcube 1.1.1 (/var/log/roundcubemail/userlogins) # failJSON: { "time": "2015-05-10T19:02:52", "match": true , "host": "1.2.3.4" } [10-May-2015 13:02:52 -0400]: <4z506z6r> Failed login for admin@example.com from 1.2.3.4 in session 4z506z6rvddstv6k7jz08hxo27 (error: 0) + +# failJSON: { "time": "2016-01-17T10:09:06", "match": true , "host": "192.0.2.2", "desc": "X-Real-IP or/and X-Forwarded-For in-between, gh-1303"} +[17-Jan-2016 11:09:06 +0200]: <j2pncet8> IMAP Error: Login failed for user@example.com from 192.0.2.2(X-Real-IP: 192.0.2.3,X-Forwarded-For: 192.0.2.4). AUTHENTICATE PLAIN: Authentication failed. in /usr/share/roundcubemail/program/lib/Roundcube/rcube_imap.php on line 197 (POST /?_task=login?_task=login&_action=login) + +# failJSON: { "time": "2005-05-19T06:07:48", "match": true , "host": "192.0.2.1", "desc": "Roundcube logged to journald instead to a local file."} +May 19 06:07:48 server roundcube[21296]: <crk9n97i> IMAP Error: Login failed for test from 192.0.2.1. AUTHENTICATE PLAIN: Authentication failed. in /usr/share/php5/Roundcube/rcube_imap.php on line 193 (POST /mail/?_task=login&_action=login) diff --git a/fail2ban/tests/files/logs/sendmail-reject b/fail2ban/tests/files/logs/sendmail-reject index 70d4dde6..44f8eb92 100644 --- a/fail2ban/tests/files/logs/sendmail-reject +++ b/fail2ban/tests/files/logs/sendmail-reject @@ -1,3 +1,5 @@ +# normal mode # filterOptions: {"mode": "normal"} + # failJSON: { "time": "2005-02-25T03:01:10", "match": true , "host": "128.68.136.133" } Feb 25 03:01:10 kismet sm-acceptingconnections[27713]: s1P819mk027713: ruleset=check_rcpt, arg1=<asservnew@freemailhost.ru>, relay=128-68-136-133.broadband.corbina.ru [128.68.136.133], reject=550 5.7.1 <asservnew@freemailhost.ru>... Relaying denied. Proper authentication required. @@ -69,20 +71,27 @@ Feb 22 14:02:44 batman sm-mta[4030]: s1MD2hsd004030: rrcs-24-73-201-194.se.biz.r # failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <arhipov@domain.com>... No such user here +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026250: <arhipov@domain.com>... No such user here # failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anatoliy@domain.com>... No such user here +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026251: <anatoliy@domain.com>... No such user here # failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <artem@domain.com>... No such user here +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026252: <artem@domain.com>... No such user here # failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anto@domain.com>... No such user here +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026252: <anto@domain.com>... No such user here + # failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anton@domain.com>... No such user here +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <davaojk25@domain.com>... No such user here # failJSON: { "time": "2004-11-03T11:35:30", "match": true , "host": "95.32.23.163" } Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: from=<davaojk25@domain.com>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=163.23.32.95.dsl-dynamic.vsi.ru [95.32.23.163] # failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026254: <anton@domain.com>... No such user here -# Different mail ID shouldn't match -# failJSON: { "match": false } -Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026255: from=<davaojk25@domain.com>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=163.23.32.95.dsl-dynamic.vsi.ru [95.32.23.163] +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026252: <anton@domain.com>... No such user here +# failJSON: { "match": false, "desc": "Different mail ID shouldn't match" } +Nov 3 11:35:30 Microsoft sendmail[26254]: rA37ZTSC026255: from=<anton@domain.com>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=163.23.32.95.dsl-dynamic.vsi.ru [95.32.23.163] + +# filterOptions: {"mode": "extra"} + +# failJSON: { "time": "2005-03-06T16:55:28", "match": true , "host": "192.0.2.194", "desc": "wrong resp. non RFC compiant (ddos prelude?), MTA-mode" } +Mar 6 16:55:28 s192-168-0-1 sm-mta[20949]: v26LtRA0020949: some-host-24.example.org [192.0.2.194] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA +# failJSON: { "time": "2005-03-07T15:04:37", "match": true , "host": "192.0.2.195", "desc": "wrong resp. non RFC compiant (ddos prelude?), MSP-mode, (may be forged)" } +Mar 7 15:04:37 s192-168-0-1 sm-mta[18624]: v27K4Vj8018624: some-host-24.example.org [192.0.2.195] (may be forged) did not issue MAIL/EXPN/VRFY/ETRN during connection to MSP-v4 diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index 0e7bfe0d..b9559359 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -1,3 +1,5 @@ +# filterOptions: [{}, {"mode": "aggressive"}] + #1 # failJSON: { "time": "2005-06-21T16:47:48", "match": true , "host": "192.030.0.6" } Jun 21 16:47:48 digital-mlhhyiqscv sshd[13709]: error: PAM: Authentication failure for myhlj1374 from 192.030.0.6 @@ -9,8 +11,14 @@ May 29 20:56:56 imago sshd[28732]: error: PAM: Authentication failure for test-i #2 # failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "194.117.26.69" } Feb 25 14:34:10 belka sshd[31602]: Failed password for invalid user ROOT from 194.117.26.69 port 50273 ssh2 +# failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1" } +Feb 25 14:34:10 belka sshd[31603]: Failed password for invalid user ROOT from aaaa:bbbb:cccc:1234::1:1 port 50273 ssh2 # failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "194.117.26.70" } Feb 25 14:34:10 belka sshd[31602]: Failed password for invalid user ROOT from 194.117.26.70 port 12345 +# failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1" } +Feb 25 14:34:10 belka sshd[31603]: Failed password for invalid user ROOT from aaaa:bbbb:cccc:1234::1:1 port 12345 +# failJSON: { "time": "2005-02-25T14:34:11", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1" } +Feb 25 14:34:11 belka sshd[31603]: Failed password for invalid user ROOT from aaaa:bbbb:cccc:1234::1:1 #3 # failJSON: { "time": "2005-01-05T01:31:41", "match": true , "host": "1.2.3.4" } @@ -19,8 +27,10 @@ Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM 1.2.3.4 Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM ::ffff:1.2.3.4 #4 -# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "211.114.51.213" } -Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 211.114.51.213 +# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "192.0.2.1", "desc": "Invalid user" } +Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 192.0.2.1 +# failJSON: { "time": "2005-07-20T14:42:12", "match": true , "host": "192.0.2.2", "desc": "Invalid user with port" } +Jul 20 14:42:12 localhost sshd[22708]: Invalid user ftp from 192.0.2.2 port 37220 #5 new filter introduced after looking at 44087D8C.9090407@bluewin.ch # yoh: added ':' after [sshd] since the case without is not really common any more @@ -105,6 +115,11 @@ May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76: 11: Bye # failJSON: { "time": "2004-09-29T16:28:02", "match": true , "host": "127.0.0.1" } Sep 29 16:28:02 spaceman sshd[16699]: Failed password for dan from 127.0.0.1 port 45416 ssh1 +# failJSON: { "match": false, "desc": "no failure, just cache mlfid (conn-id)" } +Sep 29 16:28:05 localhost sshd[16700]: Connection from 192.0.2.5 +# failJSON: { "match": false, "desc": "no failure, just covering mlfid (conn-id) forget" } +Sep 29 16:28:05 localhost sshd[16700]: Connection closed by 192.0.2.5 [preauth] + # failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1" } Sep 29 17:15:02 spaceman sshd[12946]: Failed hostbased for dan from 127.0.0.1 port 45785 ssh2: RSA 8c:e3:aa:0f:64:51:02:f7:14:79:89:3f:65:84:7c:30, client user "dan", client host "localhost.localdomain" @@ -116,6 +131,8 @@ Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 po # failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1", "desc": "Injecting while exhausting initially present {0,100} match length limits set for ruser etc" } Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX from 1.2.3.4 +# failJSON: { "time": "2004-09-29T17:15:03", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1", "desc": "Injecting while exhausting initially present {0,100} match length limits set for ruser etc" } +Sep 29 17:15:03 spaceman sshd[12946]: Failed password for user from aaaa:bbbb:cccc:1234::1:1 port 20000 ssh1: ruser XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX from 1.2.3.4 # failJSON: { "time": "2004-11-11T08:04:51", "match": true , "host": "127.0.0.1", "desc": "Injecting on username ssh 'from 10.10.1.1'@localhost" } Nov 11 08:04:51 redbamboo sshd[2737]: Failed password for invalid user from 10.10.1.1 from 127.0.0.1 port 58946 ssh2 @@ -126,6 +143,8 @@ Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from # failJSON: { "time": "2005-07-05T18:22:44", "match": true , "host": "127.0.0.1", "desc": "Failed publickey for ..." } Jul 05 18:22:44 mercury sshd[4669]: Failed publickey for graysky from 127.0.0.1 port 37954 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI +# failJSON: { "time": "2005-07-05T18:22:45", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1", "desc": "Failed publickey for ..." } +Jul 05 18:22:45 mercury sshd[4670]: Failed publickey for graysky from aaaa:bbbb:cccc:1234::1:1 port 37955 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI # failJSON: { "match": false } Nov 23 21:50:19 sshd[8148]: Disconnecting: Too many authentication failures for root [preauth] @@ -139,6 +158,8 @@ Nov 23 21:50:37 sshd[7148]: Connection closed by 61.0.0.1 [preauth] # failJSON: { "time": "2005-07-13T18:44:28", "match": true , "host": "89.24.13.192", "desc": "from gh-289" } Jul 13 18:44:28 mdop sshd[4931]: Received disconnect from 89.24.13.192: 3: com.jcraft.jsch.JSchException: Auth fail +# failJSON: { "time": "2005-01-02T01:18:41", "match": true , "host": "10.0.0.1", "desc": "space after port is optional (gh-1652)" } +Jan 2 01:18:41 host sshd[11808]: error: Received disconnect from 10.0.0.1 port 7736:3: com.jcraft.jsch.JSchException: Auth fail [preauth] # failJSON: { "time": "2004-10-01T17:27:44", "match": true , "host": "94.249.236.6", "desc": "newer format per commit 36919d9f" } Oct 1 17:27:44 localhost sshd[24077]: error: Received disconnect from 94.249.236.6: 3: com.jcraft.jsch.JSchException: Auth fail [preauth] @@ -154,7 +175,7 @@ Feb 12 04:09:21 localhost sshd[26713]: Disconnecting: Too many authentication fa # failJSON: { "match": false } Feb 12 04:09:18 localhost sshd[26713]: Connection from 115.249.163.77 port 51353 on 127.0.0.1 port 22 # failJSON: { "time": "2005-02-12T04:09:21", "match": true , "host": "115.249.163.77", "desc": "Multiline match with interface address" } -Feb 12 04:09:21 localhost sshd[26713]: Disconnecting: Too many authentication failures for root [preauth] +Feb 12 04:09:21 localhost sshd[26713]: Disconnecting: Too many authentication failures [preauth] # failJSON: { "time": "2004-11-23T21:50:37", "match": true , "host": "61.0.0.1", "desc": "New logline format as openssh 6.8 to replace prev multiline version" } Nov 23 21:50:37 myhost sshd[21810]: error: maximum authentication attempts exceeded for root from 61.0.0.1 port 49940 ssh2 [preauth] @@ -169,3 +190,56 @@ Apr 27 13:02:04 host sshd[29116]: Received disconnect from 1.2.3.4: 11: Normal S # Match sshd auth errors on OpenSUSE systems # failJSON: { "time": "2015-04-16T20:02:50", "match": true , "host": "222.186.21.217", "desc": "Authentication for user failed" } 2015-04-16T18:02:50.321974+00:00 host sshd[2716]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.217 user=root + +# filterOptions: [{"mode": "ddos"}, {"mode": "aggressive"}] + +# http://forums.powervps.com/showthread.php?t=1667 +# failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" } +Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114 + +# gh-864(1): +# failJSON: { "match": false } +Nov 24 23:46:39 host sshd[32686]: SSH: Server;Ltype: Version;Remote: 127.0.0.1-1780;Protocol: 2.0;Client: libssh2_1.4.3 +# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (1)" } +Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth] + +# gh-864(2): +# failJSON: { "match": false } +Nov 24 23:46:40 host sshd[32686]: SSH: Server;Ltype: Kex;Remote: 127.0.0.1-1780;Enc: aes128-ctr;MAC: hmac-sha1;Comp: none [preauth] +# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (2)" } +Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth] + +# gh-864(3): +# failJSON: { "match": false } +Nov 24 23:46:41 host sshd[32686]: SSH: Server;Ltype: Authname;Remote: 127.0.0.1-1780;Name: root [preauth] +# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (3)" } +Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth] + +# gh-1719: +# failJSON: { "time": "2005-03-15T09:20:57", "match": true , "host": "192.0.2.39", "desc": "Singleline for connection reset by" } +Mar 15 09:20:57 host sshd[28972]: Connection reset by 192.0.2.39 port 14282 [preauth] + + +# filterOptions: [{"mode": "extra"}, {"mode": "aggressive"}] + +# several other cases from gh-864: +# failJSON: { "time": "2004-11-25T01:34:12", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" } +Nov 25 01:34:12 srv sshd[123]: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth] +# failJSON: { "time": "2004-11-25T01:35:13", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" } +Nov 25 01:35:13 srv sshd[123]: error: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth] +# failJSON: { "time": "2004-11-25T01:35:14", "match": true , "host": "192.168.2.92", "desc": "Optional space after port" } +Nov 25 01:35:14 srv sshd[3625]: error: Received disconnect from 192.168.2.92 port 1684:14: No supported authentication methods available [preauth] + +# gh-1545: +# failJSON: { "time": "2004-11-26T13:03:29", "match": true , "host": "192.0.2.1", "desc": "No matching cipher" } +Nov 26 13:03:29 srv sshd[45]: Unable to negotiate with 192.0.2.1 port 55419: no matching cipher found. Their offer: aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,arcfour128,arcfour,3des-cbc,none [preauth] + +# gh-1117: +# failJSON: { "time": "2004-11-26T13:03:30", "match": true , "host": "192.0.2.2", "desc": "No matching key exchange method" } +Nov 26 13:03:30 srv sshd[45]: fatal: Unable to negotiate with 192.0.2.2 port 55419: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 +# failJSON: { "match": false } +Nov 26 15:03:30 host sshd[22440]: Connection from 192.0.2.3 port 39678 on 192.168.1.9 port 22 +# failJSON: { "time": "2004-11-26T15:03:31", "match": true , "host": "192.0.2.3", "desc": "Multiline - no matching key exchange method" } +Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth] +# failJSON: { "time": "2004-11-26T15:03:32", "match": true , "host": "192.0.2.3", "filter": "sshd", "desc": "Second attempt within the same connect" } +Nov 26 15:03:32 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth] diff --git a/fail2ban/tests/files/logs/sshd-ddos b/fail2ban/tests/files/logs/sshd-ddos deleted file mode 100644 index 452abbde..00000000 --- a/fail2ban/tests/files/logs/sshd-ddos +++ /dev/null @@ -1,3 +0,0 @@ -# http://forums.powervps.com/showthread.php?t=1667 -# failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" } -Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114 diff --git a/fail2ban/tests/files/logs/zzz-sshd-obsolete-multiline b/fail2ban/tests/files/logs/zzz-sshd-obsolete-multiline new file mode 100644 index 00000000..d301ad91 --- /dev/null +++ b/fail2ban/tests/files/logs/zzz-sshd-obsolete-multiline @@ -0,0 +1,2 @@ +# test sshd file: +# addFILE: "sshd" diff --git a/fail2ban/tests/files/zzz-sshd-obsolete-multiline.log b/fail2ban/tests/files/zzz-sshd-obsolete-multiline.log new file mode 100644 index 00000000..e0f6cc8a --- /dev/null +++ b/fail2ban/tests/files/zzz-sshd-obsolete-multiline.log @@ -0,0 +1,4 @@ +Apr 27 13:02:01 host sshd[29116]: Connection from 192.0.2.4 port 55555 +Apr 27 13:02:02 host sshd[29116]: User root not allowed because account is locked +Apr 27 13:02:03 host sshd[29116]: input_userauth_request: invalid user root [preauth] +Apr 27 13:02:04 host sshd[29116]: Received disconnect from 192.0.2.4: 11: Normal Shutdown, Thank you for playing [preauth] diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 986cf1f0..cb0edb06 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -43,7 +43,7 @@ from ..server.failmanager import FailManagerEmpty from ..server.ipdns import DNSUtils, IPAddr from ..server.mytime import MyTime from ..server.utils import Utils, uni_decode -from .utils import setUpMyTime, tearDownMyTime, mtimesleep, LogCaptureTestCase +from .utils import setUpMyTime, tearDownMyTime, mtimesleep, with_tmpdir, LogCaptureTestCase from .dummyjail import DummyJail TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") @@ -289,6 +289,16 @@ class BasicFilter(unittest.TestCase): ("^%Y-%m-%d-%H%M%S.%f %z **", "^Year-Month-Day-24hourMinuteSecond.Microseconds Zone offset **")) + def testGetSetLogTimeZone(self): + self.assertEqual(self.filter.getLogTimeZone(), None) + self.filter.setLogTimeZone('UTC') + self.assertEqual(self.filter.getLogTimeZone(), 'UTC') + self.filter.setLogTimeZone('UTC-0400') + self.assertEqual(self.filter.getLogTimeZone(), 'UTC-0400') + self.filter.setLogTimeZone('UTC+0200') + self.assertEqual(self.filter.getLogTimeZone(), 'UTC+0200') + self.assertRaises(ValueError, self.filter.setLogTimeZone, 'not-a-time-zone') + def testAssertWrongTime(self): self.assertRaises(AssertionError, lambda: _assert_equal_entries(self, @@ -302,7 +312,7 @@ class BasicFilter(unittest.TestCase): ## test function "_tm" works correct (returns the same as slow strftime): for i in xrange(1417512352, (1417512352 // 3600 + 3) * 3600): tm = datetime.datetime.fromtimestamp(i).strftime("%Y-%m-%d %H:%M:%S") - if _tm(i) != tm: + if _tm(i) != tm: # pragma: no cover - never reachable self.assertEqual((_tm(i), i), (tm, i)) def testWrongCharInTupleLine(self): @@ -325,6 +335,17 @@ class IgnoreIP(LogCaptureTestCase): LogCaptureTestCase.setUp(self) self.jail = DummyJail() self.filter = FileFilter(self.jail) + self.filter.ignoreSelf = False + + def testIgnoreSelfIP(self): + ipList = ("127.0.0.1",) + # test ignoreSelf is false: + for ip in ipList: + self.assertFalse(self.filter.inIgnoreIPList(ip)) + # test ignoreSelf with true: + self.filter.ignoreSelf = True + for ip in ipList: + self.assertTrue(self.filter.inIgnoreIPList(ip)) def testIgnoreIPOK(self): ipList = "127.0.0.1", "192.168.0.1", "255.255.255.255", "99.99.99.99" @@ -337,6 +358,11 @@ class IgnoreIP(LogCaptureTestCase): for ip in ipList: self.filter.addIgnoreIP(ip) self.assertFalse(self.filter.inIgnoreIPList(ip)) + if not unittest.F2B.no_network: # pragma: no cover + self.assertLogged( + 'Unable to find a corresponding IP address for 999.999.999.999', + 'Unable to find a corresponding IP address for abcdef.abcdef', + 'Unable to find a corresponding IP address for 192.168.0.', all=True) def testIgnoreIPCIDR(self): self.filter.addIgnoreIP('192.168.1.0/25') @@ -473,7 +499,7 @@ class LogFileFilterPoll(unittest.TestCase): def tearDown(self): """Call after every test case.""" - pass + super(LogFileFilterPoll, self).tearDown() #def testOpen(self): # self.filter.openLogFile(LogFile.FILENAME) @@ -791,7 +817,7 @@ class CommonMonitorTestCase(unittest.TestCase): """Wait up to `delay` sec to assure that expected failure `count` reached """ ret = Utils.wait_for( - lambda: self.filter.failManager.getFailTotal() >= self._failTotal + count and self.jail.isFilled(), + lambda: self.filter.failManager.getFailTotal() >= (self._failTotal + count) and self.jail.isFilled(), _maxWaitTime(delay)) self._failTotal += count return ret @@ -856,7 +882,7 @@ def get_monitor_failures_testcase(Filter_): #print "D: KILLING THE FILE" _killfile(self.file, self.name) #time.sleep(0.2) # Give FS time to ack the removal - pass + super(MonitorFailures, self).tearDown() def _sleep_4_poll(self): # Since FilterPoll relies on time stamps and some @@ -926,18 +952,21 @@ def get_monitor_failures_testcase(Filter_): skip=3, mode='w') self.assert_correct_last_attempt(GetFailures.FAILURES_01) - def test_move_file(self): - # if we move file into a new location while it has been open already - self.file.close() - self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, - n=14, mode='w') + def _wait4failures(self, count=2): # Poll might need more time self.assertTrue(self.isEmpty(_maxWaitTime(5)), "Queue must be empty but it is not: %s." % (', '.join([str(x) for x in self.jail.queue]))) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) - Utils.wait_for(lambda: self.filter.failManager.getFailTotal() == 2, _maxWaitTime(10)) - self.assertEqual(self.filter.failManager.getFailTotal(), 2) + Utils.wait_for(lambda: self.filter.failManager.getFailTotal() >= count, _maxWaitTime(10)) + self.assertEqual(self.filter.failManager.getFailTotal(), count) + + def test_move_file(self): + # if we move file into a new location while it has been open already + self.file.close() + self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, + n=14, mode='w') + self._wait4failures() # move aside, but leaving the handle still open... os.rename(self.name, self.name + '.bak') @@ -951,6 +980,48 @@ def get_monitor_failures_testcase(Filter_): self.assert_correct_last_attempt(GetFailures.FAILURES_01) self.assertEqual(self.filter.failManager.getFailTotal(), 6) + @with_tmpdir + def test_move_dir(self, tmp): + self.file.close() + self.filter.setMaxRetry(10) + self.filter.delLogPath(self.name) + # if we rename parent dir into a new location (simulate directory-base log rotation) + tmpsub1 = os.path.join(tmp, "1") + tmpsub2 = os.path.join(tmp, "2") + os.mkdir(tmpsub1) + self.name = os.path.join(tmpsub1, os.path.basename(self.name)) + os.close(os.open(self.name, os.O_CREAT|os.O_APPEND)); # create empty file + self.filter.addLogPath(self.name, autoSeek=False) + + self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, + skip=12, n=1, mode='w') + self.file.close() + self._wait4failures(1) + + # rotate whole directory: rename directory 1 as 2a: + os.rename(tmpsub1, tmpsub2 + 'a') + os.mkdir(tmpsub1) + self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, + skip=12, n=1, mode='w') + self.file.close() + self._wait4failures(2) + + # rotate whole directory: rename directory 1 as 2b: + os.rename(tmpsub1, tmpsub2 + 'b') + # wait a bit in-between (try to increase coverage, should find pending file for pending dir): + self.waitForTicks(2) + os.mkdir(tmpsub1) + self.waitForTicks(2) + self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, + skip=12, n=1, mode='w') + self.file.close() + self._wait4failures(3) + + # stop before tmpdir deleted (just prevents many monitor events) + self.filter.stop() + self.filter.join() + + def _test_move_into_file(self, interim_kill=False): # if we move a new file into the location of an old (monitored) file _copy_lines_between_files(GetFailures.FILENAME_01, self.name, @@ -1029,9 +1100,10 @@ def get_monitor_failures_testcase(Filter_): # total count in this test: self.assertEqual(self.filter.failManager.getFailTotal(), 12) - MonitorFailures.__name__ = "MonitorFailures<%s>(%s)" \ + cls = MonitorFailures + cls.__qualname__ = cls.__name__ = "MonitorFailures<%s>(%s)" \ % (Filter_.__name__, testclass_name) # 'tempfile') - return MonitorFailures + return cls def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover @@ -1070,7 +1142,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover if self.filter and self.filter.active: self.filter.stop() self.filter.join() # wait for the thread to terminate - pass + super(MonitorJournalFailures, self).tearDown() def _getRuntimeJournal(self): # retrieve current system journal path @@ -1211,14 +1283,16 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover fields = self.journal_fields fields.update(TEST_JOURNAL_FIELDS) journal.send(MESSAGE=l, **fields) + self.waitForTicks(1) self.waitFailTotal(6, 10) self.assertTrue(Utils.wait_for(lambda: len(self.jail) == 2, 10)) self.assertEqual(sorted([self.jail.getFailTicket().getIP(), self.jail.getFailTicket().getIP()]), ["192.0.2.1", "192.0.2.2"]) - MonitorJournalFailures.__name__ = "MonitorJournalFailures<%s>(%s)" \ + cls = MonitorJournalFailures + cls.__qualname__ = cls.__name__ = "MonitorJournalFailures<%s>(%s)" \ % (Filter_.__name__, testclass_name) - return MonitorJournalFailures + return cls class GetFailures(LogCaptureTestCase): @@ -1426,6 +1500,7 @@ class GetFailures(LogCaptureTestCase): ('no', output_no), ('warn', output_yes) ): + self.pruneLog("[test-phase useDns=%s]" % useDns) jail = DummyJail() filter_ = FileFilter(jail, useDns=useDns) filter_.active = True @@ -1459,8 +1534,8 @@ class GetFailures(LogCaptureTestCase): output = [("192.0.43.10", 2, 1124013599.0), ("192.0.43.11", 1, 1124013598.0)] self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False) - self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") self.filter.setMaxLines(100) + self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") self.filter.setMaxRetry(1) self.filter.getFailures(GetFailures.FILENAME_MULTILINE) @@ -1477,9 +1552,9 @@ class GetFailures(LogCaptureTestCase): def testGetFailuresMultiLineIgnoreRegex(self): output = [("192.0.43.10", 2, 1124013599.0)] self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False) + self.filter.setMaxLines(100) self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") self.filter.addIgnoreRegex("rsync error: Received SIGINT") - self.filter.setMaxLines(100) self.filter.setMaxRetry(1) self.filter.getFailures(GetFailures.FILENAME_MULTILINE) @@ -1493,9 +1568,9 @@ class GetFailures(LogCaptureTestCase): ("192.0.43.11", 1, 1124013598.0), ("192.0.43.15", 1, 1124013598.0)] self.filter.addLogPath(GetFailures.FILENAME_MULTILINE, autoSeek=False) + self.filter.setMaxLines(100) self.filter.addFailRegex("^.*rsyncd\[(?P<pid>\d+)\]: connect from .+ \(<HOST>\)$<SKIPLINES>^.+ rsyncd\[(?P=pid)\]: rsync error: .*$") self.filter.addFailRegex("^.* sendmail\[.*, msgid=<(?P<msgid>[^>]+).*relay=\[<HOST>\].*$<SKIPLINES>^.+ spamd: result: Y \d+ .*,mid=<(?P=msgid)>(,bayes=[.\d]+)?(,autolearn=\S+)?\s*$") - self.filter.setMaxLines(100) self.filter.setMaxRetry(1) self.filter.getFailures(GetFailures.FILENAME_MULTILINE) diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py index 0f384a7b..5f0a447a 100644 --- a/fail2ban/tests/samplestestcase.py +++ b/fail2ban/tests/samplestestcase.py @@ -31,6 +31,7 @@ import re import sys import time import unittest +from ..server.failregex import Regex from ..server.filter import Filter from ..client.filterreader import FilterReader from .utils import setUpMyTime, tearDownMyTime, CONFIG_DIR @@ -38,22 +39,23 @@ from .utils import setUpMyTime, tearDownMyTime, CONFIG_DIR TEST_CONFIG_DIR = os.path.join(os.path.dirname(__file__), "config") TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") +# regexp to test greedy catch-all should be not-greedy: +RE_HOST = Regex._resolveHostTag('<HOST>') +RE_WRONG_GREED = re.compile(r'\.[+\*](?!\?)[^\$\^]*' + re.escape(RE_HOST) + r'.*(?:\.[+\*].*|[^\$])$') + class FilterSamplesRegex(unittest.TestCase): def setUp(self): """Call before every test case.""" super(FilterSamplesRegex, self).setUp() - self.filter = Filter(None) - self.filter.returnRawHost = True - self.filter.checkAllRegex = True - self.filter.checkFindTime = False - self.filter.active = True - + self._filters = dict() + self._filterTests = None setUpMyTime() def tearDown(self): """Call after every test case.""" + super(FilterSamplesRegex, self).tearDown() tearDownMyTime() def testFiltersPresent(self): @@ -64,12 +66,35 @@ class FilterSamplesRegex(unittest.TestCase): >= 10, "Expected more FilterSampleRegexs tests") + def testReWrongGreedyCatchAll(self): + """Tests regexp RE_WRONG_GREED is intact (positive/negative)""" + self.assertTrue( + RE_WRONG_GREED.search('greedy .* test' + RE_HOST + ' test not hard-anchored')) + self.assertTrue( + RE_WRONG_GREED.search('greedy .+ test' + RE_HOST + ' test vary .* anchored$')) + self.assertFalse( + RE_WRONG_GREED.search('greedy .* test' + RE_HOST + ' test no catch-all, hard-anchored$')) + self.assertFalse( + RE_WRONG_GREED.search('non-greedy .*? test' + RE_HOST + ' test not hard-anchored')) + self.assertFalse( + RE_WRONG_GREED.search('non-greedy .+? test' + RE_HOST + ' test vary catch-all .* anchored$')) -def testSampleRegexsFactory(name, basedir): - def testFilter(self): - # Check filter exists - filterConf = FilterReader(name, "jail", {}, + def _readFilter(self, fltName, name, basedir, opts=None): + # Check filter with this option combination was already used: + flt = self._filters.get(fltName) + if flt: + return flt + # First time: + flt = Filter(None) + flt.returnRawHost = True + flt.checkAllRegex = True + flt.checkFindTime = False + flt.active = True + # Read filter: + if opts is None: opts = dict() + opts = opts.copy() + filterConf = FilterReader(name, "jail", opts, basedir=basedir, share_config=unittest.F2B.share_config) self.assertEqual(filterConf.getFile(), name) self.assertEqual(filterConf.getJailName(), "jail") @@ -81,81 +106,158 @@ def testSampleRegexsFactory(name, basedir): optval = opt[3] elif opt[0] == 'set': optval = [opt[3]] - else: - continue + else: # pragma: no cover - unexpected + self.fail('Unexpected config-token %r in stream' % (opt,)) for optval in optval: - if opt[2] == "addfailregex": - self.filter.addFailRegex(optval) + if opt[2] == "prefregex": + flt.prefRegex = optval + elif opt[2] == "addfailregex": + flt.addFailRegex(optval) elif opt[2] == "addignoreregex": - self.filter.addIgnoreRegex(optval) + flt.addIgnoreRegex(optval) elif opt[2] == "maxlines": - self.filter.setMaxLines(optval) + flt.setMaxLines(optval) elif opt[2] == "datepattern": - self.filter.setDatePattern(optval) + flt.setDatePattern(optval) + + # test regexp contains greedy catch-all before <HOST>, that is + # not hard-anchored at end or has not precise sub expression after <HOST>: + regexList = flt.getFailRegex() + for fr in regexList: + if RE_WRONG_GREED.search(fr): # pragma: no cover + raise AssertionError("Following regexp of \"%s\" contains greedy catch-all before <HOST>, " + "that is not hard-anchored at end or has not precise sub expression after <HOST>:\n%s" % + (fltName, str(fr).replace(RE_HOST, '<HOST>'))) + # Cache within used filter combinations and return: + flt = [flt, set()] + self._filters[fltName] = flt + return flt + +def testSampleRegexsFactory(name, basedir): + def testFilter(self): self.assertTrue( os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", name)), "No sample log file available for '%s' filter" % name) + + filenames = [name] + regexsUsedRe = set() + + # process each test-file (note: array filenames can grow during processing): + i = 0 + while i < len(filenames): + filename = filenames[i]; i += 1; + logFile = fileinput.FileInput(os.path.join(TEST_FILES_DIR, "logs", + filename)) + + for line in logFile: + jsonREMatch = re.match("^#+ ?(failJSON|filterOptions|addFILE):(.+)$", line) + if jsonREMatch: + try: + faildata = json.loads(jsonREMatch.group(2)) + # filterOptions - dict in JSON to control filter options (e. g. mode, etc.): + if jsonREMatch.group(1) == 'filterOptions': + # following lines with another filter options: + self._filterTests = [] + for opts in (faildata if isinstance(faildata, list) else [faildata]): + # unique filter name (using options combination): + self.assertTrue(isinstance(opts, dict)) + fltName = opts.get('filterName') + if not fltName: fltName = str(opts) if opts else '' + fltName = name + fltName + # read it: + flt = self._readFilter(fltName, name, basedir, opts=opts) + self._filterTests.append((fltName, flt)) + continue + # addFILE - filename to "include" test-files should be additionally parsed: + if jsonREMatch.group(1) == 'addFILE': + filenames.append(faildata) + continue + # failJSON - faildata contains info of the failure to check it. + except ValueError as e: # pragma: no cover - we've valid json's + raise ValueError("%s: %s:%i" % + (e, logFile.filename(), logFile.filelineno())) + line = next(logFile) + elif line.startswith("#") or not line.strip(): + continue + else: # pragma: no cover - normally unreachable + faildata = {} + + # if filter options was not yet specified: + if not self._filterTests: + fltName = name + flt = self._readFilter(fltName, name, basedir, opts=None) + self._filterTests = [(fltName, flt)] + + # process line using several filter options (if specified in the test-file): + for fltName, flt in self._filterTests: + flt, regexsUsedIdx = flt + regexList = flt.getFailRegex() + + try: + ret = flt.processLine(line) + if not ret: + # Bypass if filter constraint specified: + if faildata.get('filter') and name != faildata.get('filter'): + continue + # Check line is flagged as none match + self.assertFalse(faildata.get('match', True), + "Line not matched when should have") + continue + + failregex, fid, fail2banTime, fail = ret[0] + # Bypass no failure helpers-regexp: + if not faildata.get('match', False) and (fid is None or fail.get('nofail')): + regexsUsedIdx.add(failregex) + regexsUsedRe.add(regexList[failregex]) + continue + + # Check line is flagged to match + self.assertTrue(faildata.get('match', False), + "Line matched when shouldn't have") + self.assertEqual(len(ret), 1, + "Multiple regexs matched %r" % (map(lambda x: x[0], ret))) + + # Verify match captures (at least fid/host) and timestamp as expected + for k, v in faildata.iteritems(): + if k not in ("time", "match", "desc", "filter"): + fv = fail.get(k, None) + # Fallback for backwards compatibility (previously no fid, was host only): + if k == "host" and fv is None: + fv = fid + self.assertEqual(fv, v) + + t = faildata.get("time", None) + try: + jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S") + except ValueError: + jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S.%f") + + jsonTime = time.mktime(jsonTimeLocal.timetuple()) + + jsonTime += jsonTimeLocal.microsecond / 1000000 + + self.assertEqual(fail2banTime, jsonTime, + "UTC Time mismatch %s (%s) != %s (%s) (diff %.3f seconds)" % + (fail2banTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(fail2banTime)), + jsonTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(jsonTime)), + fail2banTime - jsonTime) ) + + regexsUsedIdx.add(failregex) + regexsUsedRe.add(regexList[failregex]) + except AssertionError as e: # pragma: no cover + raise AssertionError("%s: %s on: %s:%i, line:\n%s" % ( + fltName, e, logFile.filename(), logFile.filelineno(), line)) - logFile = fileinput.FileInput( - os.path.join(TEST_FILES_DIR, "logs", name)) - - regexsUsed = set() - for line in logFile: - jsonREMatch = re.match("^# ?failJSON:(.+)$", line) - if jsonREMatch: - try: - faildata = json.loads(jsonREMatch.group(1)) - except ValueError as e: - raise ValueError("%s: %s:%i" % - (e, logFile.filename(), logFile.filelineno())) - line = next(logFile) - elif line.startswith("#") or not line.strip(): - continue - else: - faildata = {} - - ret = self.filter.processLine(line) - if not ret: - # Check line is flagged as none match - self.assertFalse(faildata.get('match', True), - "Line not matched when should have: %s:%i %r" % - (logFile.filename(), logFile.filelineno(), line)) - elif ret: - # Check line is flagged to match - self.assertTrue(faildata.get('match', False), - "Line matched when shouldn't have: %s:%i %r" % - (logFile.filename(), logFile.filelineno(), line)) - self.assertEqual(len(ret), 1, "Multiple regexs matched %r - %s:%i" % - (map(lambda x: x[0], ret),logFile.filename(), logFile.filelineno())) - - # Verify timestamp and host as expected - failregex, host, fail2banTime, lines, fail = ret[0] - self.assertEqual(host, faildata.get("host", None)) - - t = faildata.get("time", None) - try: - jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S") - except ValueError: - jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S.%f") - - jsonTime = time.mktime(jsonTimeLocal.timetuple()) - - jsonTime += jsonTimeLocal.microsecond / 1000000 - - self.assertEqual(fail2banTime, jsonTime, - "UTC Time mismatch fail2ban %s (%s) != failJson %s (%s) (diff %.3f seconds) on: %s:%i %r:" % - (fail2banTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(fail2banTime)), - jsonTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(jsonTime)), - fail2banTime - jsonTime, logFile.filename(), logFile.filelineno(), line ) ) - - regexsUsed.add(failregex) - - for failRegexIndex, failRegex in enumerate(self.filter.getFailRegex()): - self.assertTrue( - failRegexIndex in regexsUsed, - "Regex for filter '%s' has no samples: %i: %r" % - (name, failRegexIndex, failRegex)) + # check missing samples for regex using each filter-options combination: + for fltName, flt in self._filters.iteritems(): + flt, regexsUsedIdx = flt + regexList = flt.getFailRegex() + for failRegexIndex, failRegex in enumerate(regexList): + self.assertTrue( + failRegexIndex in regexsUsedIdx or failRegex in regexsUsedRe, + "%s: Regex has no samples: %i: %r" % + (fltName, failRegexIndex, failRegex)) return testFilter diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index aada699c..68b9951c 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -38,7 +38,9 @@ from ..server.server import Server from ..server.ipdns import IPAddr from ..server.jail import Jail from ..server.jailthread import JailThread +from ..server.ticket import BanTicket from ..server.utils import Utils +from .dummyjail import DummyJail from .utils import LogCaptureTestCase from ..helpers import getLogger, PREFER_ENC from .. import version @@ -77,7 +79,7 @@ class TransmitterBase(unittest.TestCase): """Call after every test case.""" # stop jails, etc. self.server.quit() - #super(TransmitterBase, self).tearDown() + super(TransmitterBase, self).tearDown() def setGetTest(self, cmd, inValue, outValue=(None,), outCode=0, jail=None, repr_=False): """Process set/get commands and compare both return values @@ -309,6 +311,10 @@ class Transmitter(TransmitterBase): "datepattern", "TAI64N", (None, "TAI64N"), jail=self.jailName) self.setGetTestNOK("datepattern", "%Cat%a%%%g", jail=self.jailName) + def testLogTimeZone(self): + self.setGetTest("logtimezone", "UTC+0400", "UTC+0400", jail=self.jailName) + self.setGetTestNOK("logtimezone", "not-a-time-zone", jail=self.jailName) + def testJailUseDNS(self): self.setGetTest("usedns", "yes", jail=self.jailName) self.setGetTest("usedns", "warn", jail=self.jailName) @@ -447,6 +453,16 @@ class Transmitter(TransmitterBase): self.transm.proceed(["set", self.jailName, "delignoreip", value]), (0, [value])) + self.assertEqual( + self.transm.proceed(["get", self.jailName, "ignoreself"]), + (0, True)) + self.assertEqual( + self.transm.proceed(["set", self.jailName, "ignoreself", False]), + (0, False)) + self.assertEqual( + self.transm.proceed(["get", self.jailName, "ignoreself"]), + (0, False)) + def testJailIgnoreCommand(self): self.setGetTest("ignorecommand", "bin ", jail=self.jailName) @@ -992,9 +1008,10 @@ class LoggingTests(LogCaptureTestCase): badThread = _BadThread() badThread.start() badThread.join() - self.assertLogged("Unhandled exception") + self.assertTrue( Utils.wait_for( lambda: len(x) and self._is_logged("Unhandled exception"), 3) ) finally: sys.__excepthook__ = prev_exchook + self.assertLogged("Unhandled exception") self.assertEqual(len(x), 1) self.assertEqual(x[0][0], RuntimeError) @@ -1061,16 +1078,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): action.start() # test ban ip4 : logSys.debug('# === ban-ipv4 ==='); self.pruneLog() - action.ban({'ip': IPAddr('192.0.2.1')}) + action.ban({'ip': IPAddr('192.0.2.1'), 'family': 'inet4'}) # test unban ip4 : logSys.debug('# === unban ipv4 ==='); self.pruneLog() - action.unban({'ip': IPAddr('192.0.2.1')}) + action.unban({'ip': IPAddr('192.0.2.1'), 'family': 'inet4'}) # test ban ip6 : logSys.debug('# === ban ipv6 ==='); self.pruneLog() - action.ban({'ip': IPAddr('2001:DB8::')}) + action.ban({'ip': IPAddr('2001:DB8::'), 'family': 'inet6'}) # test unban ip6 : logSys.debug('# === unban ipv6 ==='); self.pruneLog() - action.unban({'ip': IPAddr('2001:DB8::')}) + action.unban({'ip': IPAddr('2001:DB8::'), 'family': 'inet6'}) # test stop : logSys.debug('# === stop ==='); self.pruneLog() action.stop() @@ -1169,17 +1186,50 @@ class ServerConfigReaderTests(LogCaptureTestCase): # 'start', 'stop' - should be found (logged) on action start/stop, # etc. testJailsActions = ( + # dummy -- + ('j-dummy', 'dummy[name=%(__name__)s, init="==", target="/tmp/fail2ban.dummy"]', { + 'ip4': ('family: inet4',), 'ip6': ('family: inet6',), + 'start': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- started"`', + ), + 'flush': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- clear all"`', + ), + 'stop': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- stopped"`', + ), + 'ip4-check': (), + 'ip6-check': (), + 'ip4-ban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 192.0.2.1 (family: inet4)"`', + ), + 'ip4-unban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 192.0.2.1 (family: inet4)"`', + ), + 'ip6-ban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- banned 2001:db8:: (family: inet6)"`', + ), + 'ip6-unban': ( + '`echo "[j-dummy] dummy /tmp/fail2ban.dummy -- unbanned 2001:db8:: (family: inet6)"`', + ), + }), # iptables-multiport -- ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain="INPUT"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'start': ( + 'ip4-start': ( "`iptables -w -N f2b-j-w-iptables-mp`", "`iptables -w -A f2b-j-w-iptables-mp -j RETURN`", "`iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + ), + 'ip6-start': ( "`ip6tables -w -N f2b-j-w-iptables-mp`", "`ip6tables -w -A f2b-j-w-iptables-mp -j RETURN`", "`ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables-mp`", + "`ip6tables -w -F f2b-j-w-iptables-mp`", + ), 'stop': ( "`iptables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", "`iptables -w -F f2b-j-w-iptables-mp`", @@ -1210,14 +1260,20 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-allports -- ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain="INPUT"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'start': ( + 'ip4-start': ( "`iptables -w -N f2b-j-w-iptables-ap`", "`iptables -w -A f2b-j-w-iptables-ap -j RETURN`", "`iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + ), + 'ip6-start': ( "`ip6tables -w -N f2b-j-w-iptables-ap`", "`ip6tables -w -A f2b-j-w-iptables-ap -j RETURN`", "`ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables-ap`", + "`ip6tables -w -F f2b-j-w-iptables-ap`", + ), 'stop': ( "`iptables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", "`iptables -w -F f2b-j-w-iptables-ap`", @@ -1248,12 +1304,18 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-ipset-proto6 -- ('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', { 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), - 'start': ( + 'ip4-start': ( "`ipset create f2b-j-w-iptables-ipset hash:ip timeout 600`", "`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( "`ipset create f2b-j-w-iptables-ipset6 hash:ip timeout 600 family inet6`", "`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", ), + 'flush': ( + "`ipset flush f2b-j-w-iptables-ipset`", + "`ipset flush f2b-j-w-iptables-ipset6`", + ), 'stop': ( "`iptables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset`", @@ -1280,12 +1342,18 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-ipset-proto6-allports -- ('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, bantime="10m", chain="INPUT"]', { 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), - 'start': ( + 'ip4-start': ( "`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 600`", "`iptables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( "`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 600 family inet6`", "`ip6tables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", ), + 'flush': ( + "`ipset flush f2b-j-w-iptables-ipset-ap`", + "`ipset flush f2b-j-w-iptables-ipset-ap6`", + ), 'stop': ( "`iptables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap`", @@ -1312,14 +1380,20 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables -- ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'start': ( + 'ip4-start': ( "`iptables -w -N f2b-j-w-iptables`", "`iptables -w -A f2b-j-w-iptables -j RETURN`", "`iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + ), + 'ip6-start': ( "`ip6tables -w -N f2b-j-w-iptables`", "`ip6tables -w -A f2b-j-w-iptables -j RETURN`", "`ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables`", + "`ip6tables -w -F f2b-j-w-iptables`", + ), 'stop': ( "`iptables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", "`iptables -w -F f2b-j-w-iptables`", @@ -1350,14 +1424,20 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-new -- ('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), - 'start': ( + 'ip4-start': ( "`iptables -w -N f2b-j-w-iptables-new`", "`iptables -w -A f2b-j-w-iptables-new -j RETURN`", "`iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + ), + 'ip6-start': ( "`ip6tables -w -N f2b-j-w-iptables-new`", "`ip6tables -w -A f2b-j-w-iptables-new -j RETURN`", "`ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", ), + 'flush': ( + "`iptables -w -F f2b-j-w-iptables-new`", + "`ip6tables -w -F f2b-j-w-iptables-new`", + ), 'stop': ( "`iptables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", "`iptables -w -F f2b-j-w-iptables-new`", @@ -1388,8 +1468,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-xt_recent-echo -- ('j-w-iptables-xtre', 'iptables-xt_recent-echo[name=%(__name__)s, bantime="10m", chain="INPUT"]', { 'ip4': ('`iptables ', '/f2b-j-w-iptables-xtre`'), 'ip6': ('`ip6tables ', '/f2b-j-w-iptables-xtre6`'), - 'start': ( + 'ip4-start': ( "`if [ `id -u` -eq 0 ];then iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + ), + 'ip6-start': ( "`if [ `id -u` -eq 0 ];then ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", ), 'stop': ( @@ -1418,7 +1500,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): ), }), # pf default -- multiport on default port (tag <port> set in jail.conf, but not in this test case) - ('j-w-pf', 'pf[name=%(__name__)s]', { + ('j-w-pf', 'pf[name=%(__name__)s, actionstart_on_demand=false]', { 'ip4': (), 'ip6': (), 'start': ( '`echo "table <f2b-j-w-pf> persist counters" | pfctl -f-`', @@ -1455,13 +1537,14 @@ class ServerConfigReaderTests(LogCaptureTestCase): 'ip6-ban': ("`pfctl -t f2b-j-w-pf-mp -T add 2001:db8::`",), 'ip6-unban': ("`pfctl -t f2b-j-w-pf-mp -T delete 2001:db8::`",), }), - # pf allports -- - ('j-w-pf-ap', 'pf[actiontype=<allports>][name=%(__name__)s]', { + # pf allports -- test additionally "actionstart_on_demand" was set to true + ('j-w-pf-ap', 'pf[actiontype=<allports>, actionstart_on_demand=true][name=%(__name__)s]', { 'ip4': (), 'ip6': (), - 'start': ( + 'ip4-start': ( '`echo "table <f2b-j-w-pf-ap> persist counters" | pfctl -f-`', '`echo "block proto tcp from <f2b-j-w-pf-ap> to any" | pfctl -f-`', ), + 'ip6-start': (), # the same as ipv4 'stop': ( '`pfctl -sr 2>/dev/null | grep -v f2b-j-w-pf-ap | pfctl -f-`', '`pfctl -t f2b-j-w-pf-ap -T flush`', @@ -1477,10 +1560,12 @@ class ServerConfigReaderTests(LogCaptureTestCase): # firewallcmd-multiport -- ('j-w-fwcmd-mp', 'firewallcmd-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain="INPUT"]', { 'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'), - 'start': ( + 'ip4-start': ( "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", "`firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", + ), + 'ip6-start': ( "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-mp`", "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-mp 1000 -j RETURN`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT 0 -m conntrack --ctstate NEW -p tcp -m multiport --dports http,https -j f2b-j-w-fwcmd-mp`", @@ -1515,10 +1600,12 @@ class ServerConfigReaderTests(LogCaptureTestCase): # firewallcmd-allports -- ('j-w-fwcmd-ap', 'firewallcmd-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain="INPUT"]', { 'ip4': (' ipv4 ', 'icmp-port-unreachable'), 'ip6': (' ipv6 ', 'icmp6-port-unreachable'), - 'start': ( + 'ip4-start': ( "`firewall-cmd --direct --add-chain ipv4 filter f2b-j-w-fwcmd-ap`", "`firewall-cmd --direct --add-rule ipv4 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`", "`firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -j f2b-j-w-fwcmd-ap`", + ), + 'ip6-start': ( "`firewall-cmd --direct --add-chain ipv6 filter f2b-j-w-fwcmd-ap`", "`firewall-cmd --direct --add-rule ipv6 filter f2b-j-w-fwcmd-ap 1000 -j RETURN`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT 0 -j f2b-j-w-fwcmd-ap`", @@ -1553,9 +1640,11 @@ class ServerConfigReaderTests(LogCaptureTestCase): # firewallcmd-ipset -- ('j-w-fwcmd-ipset', 'firewallcmd-ipset[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="INPUT"]', { 'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',), - 'start': ( + 'ip4-start': ( "`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 600`", "`firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", + ), + 'ip6-start': ( "`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 600`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", ), @@ -1601,6 +1690,10 @@ class ServerConfigReaderTests(LogCaptureTestCase): jails = server._Server__jails + tickets = { + 'ip4': BanTicket('192.0.2.1'), + 'ip6': BanTicket('2001:DB8::'), + } for jail, act, tests in testJailsActions: # print(jail, jails[jail]) for a in jails[jail].actions: @@ -1614,27 +1707,43 @@ class ServerConfigReaderTests(LogCaptureTestCase): # test start : self.pruneLog('# === start ===') action.start() - self.assertLogged(*tests['start'], all=True) + if tests.get('start'): + self.assertLogged(*tests['start'], all=True) + else: + self.assertNotLogged(*tests['ip4-start']+tests['ip6-start'], all=True) + ainfo = { + 'ip4': _actions.Actions.ActionInfo(tickets['ip4'], jails[jail]), + 'ip6': _actions.Actions.ActionInfo(tickets['ip6'], jails[jail]), + } # test ban ip4 : self.pruneLog('# === ban-ipv4 ===') - action.ban({'ip': IPAddr('192.0.2.1')}) + action.ban(ainfo['ip4']) + if tests.get('ip4-start'): self.assertLogged(*tests['ip4-start'], all=True) + if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True) self.assertLogged(*tests['ip4-check']+tests['ip4-ban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test unban ip4 : self.pruneLog('# === unban ipv4 ===') - action.unban({'ip': IPAddr('192.0.2.1')}) + action.unban(ainfo['ip4']) self.assertLogged(*tests['ip4-check']+tests['ip4-unban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test ban ip6 : self.pruneLog('# === ban ipv6 ===') - action.ban({'ip': IPAddr('2001:DB8::')}) + action.ban(ainfo['ip6']) + if tests.get('ip6-start'): self.assertLogged(*tests['ip6-start'], all=True) + if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True) self.assertLogged(*tests['ip6-check']+tests['ip6-ban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) # test unban ip6 : self.pruneLog('# === unban ipv6 ===') - action.unban({'ip': IPAddr('2001:DB8::')}) + action.unban(ainfo['ip6']) self.assertLogged(*tests['ip6-check']+tests['ip6-unban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) + # test flush for actions should supported this: + if tests.get('flush'): + self.pruneLog('# === flush ===') + action.flush() + self.assertLogged(*tests['flush'], all=True) # test stop : self.pruneLog('# === stop ===') action.stop() @@ -1643,10 +1752,11 @@ class ServerConfigReaderTests(LogCaptureTestCase): def _executeMailCmd(self, realCmd, timeout=60): # replace pipe to mail with pipe to cat: realCmd = re.sub(r'\)\s*\|\s*mail\b([^\n]*)', - r' echo mail \1 ) | cat', realCmd) - # replace abuse retrieving (possible no-network): - realCmd = re.sub(r'[^\n]+\bADDRESSES=\$\(dig\s[^\n]+', - 'ADDRESSES="abuse-1@abuse-test-server, abuse-2@abuse-test-server"', realCmd) + r') | cat; printf "\\n... | "; echo mail \1', realCmd) + # replace abuse retrieving (possible no-network), just replace first occurrence of 'dig...': + realCmd = re.sub(r'\bADDRESSES=\$\(dig\s[^\n]+', + lambda m: 'ADDRESSES="abuse-1@abuse-test-server, abuse-2@abuse-test-server"', + realCmd, 1) # execute action: return _actions.CommandAction.executeCmd(realCmd, timeout=timeout) @@ -1675,18 +1785,29 @@ class ServerConfigReaderTests(LogCaptureTestCase): # complain -- ('j-complain-abuse', 'complain[' - 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s",' + + 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s \'Hostname: <ip-host>, family: <family>\' - ",' + + # test reverse ip: + 'debug=1,' + # 2 logs to test grep from multiple logs: 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' ']', { 'ip4-ban': ( + # test reverse ip: + 'try to resolve 10.124.142.87.abuse-contacts.abusix.org', 'Lines containing failures of 87.142.124.10 (max 2)', 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', # both abuse mails should be separated with space: - 'mail -s Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server', + 'mail -s Hostname: test-host, family: inet4 - Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server', + ), + 'ip6-ban': ( + # test reverse ip: + 'try to resolve 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.abuse-contacts.abusix.org', + 'Lines containing failures of 2001:db8::1 (max 2)', + # both abuse mails should be separated with space: + 'mail -s Hostname: test-host, family: inet6 - Abuse from 2001:db8::1 abuse-1@abuse-test-server abuse-2@abuse-test-server', ), }), ) @@ -1708,6 +1829,9 @@ class ServerConfigReaderTests(LogCaptureTestCase): jails = server._Server__jails + ipv4 = IPAddr('87.142.124.10') + ipv6 = IPAddr('2001:db8::1'); + dmyjail = DummyJail() for jail, act, tests in testJailsActions: # print(jail, jails[jail]) for a in jails[jail].actions: @@ -1718,8 +1842,11 @@ class ServerConfigReaderTests(LogCaptureTestCase): # wrap default command processor: action.executeCmd = self._executeMailCmd # test ban : - self.pruneLog('# === ban ===') - action.ban({'ip': IPAddr('87.142.124.10'), - 'failures': 100, - }) - self.assertLogged(*tests['ip4-ban'], all=True) + for (test, ip) in (('ip4-ban', ipv4), ('ip6-ban', ipv6)): + if not tests.get(test): continue + self.pruneLog('# === %s ===' % test) + ticket = BanTicket(ip) + ticket.setAttempt(100) + ticket = _actions.Actions.ActionInfo(ticket, dmyjail) + action.ban(ticket) + self.assertLogged(*tests[test], all=True) diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index ae543e3d..9f92c26b 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -37,7 +37,7 @@ import unittest from cStringIO import StringIO from functools import wraps -from ..helpers import getLogger, str2LogLevel, getVerbosityFormat +from ..helpers import getLogger, str2LogLevel, getVerbosityFormat, uni_decode from ..server.ipdns import DNSUtils from ..server.mytime import MyTime from ..server.utils import Utils @@ -262,17 +262,25 @@ def initTests(opts): # persistently set time zone to CET (used in zone-related test-cases), # yoh: we need to adjust TZ to match the one used by Cyril so all the timestamps match - os.environ['TZ'] = 'Europe/Zurich' + # This offset corresponds to Europe/Zurich timezone. Specifying it + # explicitly allows to avoid requiring tzdata package to be installed during + # testing. See https://bugs.debian.org/855920 for more information + os.environ['TZ'] = 'CET-01CEST-02,M3.5.0,M10.5.0' time.tzset() # set alternate now for time related test cases: MyTime.setAlternateNow(TEST_NOW) # precache all invalid ip's (TEST-NET-1, ..., TEST-NET-3 according to RFC 5737): c = DNSUtils.CACHE_ipToName - for i in xrange(255): + # increase max count and max time (too many entries, long time testing): + c.setOptions(maxCount=10000, maxTime=5*60) + for i in xrange(256): c.set('192.0.2.%s' % i, None) c.set('198.51.100.%s' % i, None) c.set('203.0.113.%s' % i, None) + c.set('2001:db8::%s' %i, 'test-host') + # some legal ips used in our test cases (prevent slow dns-resolving and failures if will be changed later): + c.set('87.142.124.10', 'test-host') if unittest.F2B.no_network: # pragma: no cover # precache all wrong dns to ip's used in test cases: c = DNSUtils.CACHE_nameToIp @@ -526,12 +534,21 @@ if True: ## if not hasattr(unittest.TestCase, 'assertIn'): _org_setUp = unittest.TestCase.setUp def _customSetUp(self): # print('=='*10, self) - if unittest.F2B.log_level <= logging.DEBUG: # so if DEBUG etc -- show them (and log it in travis)! - print("") + # so if DEBUG etc -- show them (and log it in travis)! + if unittest.F2B.log_level <= logging.DEBUG: # pragma: no cover + sys.stderr.write("\n") logSys.debug('='*10 + ' %s ' + '='*20, self.id()) _org_setUp(self) + if unittest.F2B.verbosity > 2: # pragma: no cover + self.__startTime = time.time() + +_org_tearDown = unittest.TestCase.tearDown +def _customTearDown(self): + if unittest.F2B.verbosity > 2: # pragma: no cover + sys.stderr.write(" %.3fs -- " % (time.time() - self.__startTime,)) unittest.TestCase.setUp = _customSetUp +unittest.TestCase.tearDown = _customTearDown class LogCaptureTestCase(unittest.TestCase): diff --git a/files/debian-initd b/files/debian-initd index 0d2bed9a..d6660215 100755 --- a/files/debian-initd +++ b/files/debian-initd @@ -22,7 +22,7 @@ # rename this file: (sudo) mv /etc/init.d/fail2ban.init /etc/init.d/fail2ban # same with the logrotate file: (sudo) mv /etc/logrotate.d/fail2ban.logrotate /etc/logrotate.d/fail2ban # -PATH=/usr/sbin:/usr/bin:/sbin:/bin +PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin DESC="authentication failure monitor" NAME=fail2ban diff --git a/files/gentoo-initd b/files/gentoo-initd index c5b5b702..c4d8675c 100755 --- a/files/gentoo-initd +++ b/files/gentoo-initd @@ -41,7 +41,7 @@ start() { stop() { ebegin "Stopping fail2ban" - start-stop-daemon --stop --pidfile /var/run/fail2ban/fail2ban.pid \ + start-stop-daemon --stop --pidfile /var/run/fail2ban/fail2ban.pid --retry 30 \ -- ${FAIL2BAN} stop eend $? "Failed to stop fail2ban" } diff --git a/man/fail2ban-client.1 b/man/fail2ban-client.1 index 281d8f5b..08de0af4 100644 --- a/man/fail2ban-client.1 +++ b/man/fail2ban-client.1 @@ -1,5 +1,5 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-CLIENT "1" "September 2016" "fail2ban-client v0.10.0a2" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-CLIENT "1" "August 2017" "fail2ban-client v0.10.0a2" "User Commands" .SH NAME fail2ban-client \- configure and control the server .SH SYNOPSIS @@ -29,6 +29,9 @@ logging level \fB\-d\fR dump configuration. For debugging .TP +\fB\-t\fR, \fB\-\-test\fR +test configuration (can be also specified with start parameters) +.TP \fB\-i\fR interactive mode .TP @@ -53,6 +56,9 @@ start server in async mode (for internal usage only, don't read configuration) \fB\-\-timeout\fR timeout to wait for the server (for internal usage only, don't read configuration) .TP +\fB\-\-str2sec\fR <STRING> +convert time abbreviation format to seconds +.TP \fB\-h\fR, \fB\-\-help\fR display this help message .TP @@ -188,6 +194,10 @@ JAIL CONFIGURATION \fBset <JAIL> idle on|off\fR sets the idle state of <JAIL> .TP +\fBset <JAIL> ignoreself true|false\fR +allows the ignoring of own IP +addresses +.TP \fBset <JAIL> addignoreip <IP>\fR adds <IP> to the ignore list of <JAIL> @@ -332,6 +342,10 @@ for <JAIL> gets the journal filter match for <JAIL> .TP +\fBget <JAIL> ignoreself\fR +gets the current value of the +ignoring the own IP addresses +.TP \fBget <JAIL> ignoreip\fR gets the list of ignored IP addresses for <JAIL> diff --git a/man/fail2ban-regex.1 b/man/fail2ban-regex.1 index 44e13c86..94122aa4 100644 --- a/man/fail2ban-regex.1 +++ b/man/fail2ban-regex.1 @@ -1,5 +1,5 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-REGEX "1" "September 2016" "fail2ban-regex 0.10.0a2" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-REGEX "1" "August 2017" "fail2ban-regex 0.10.0a2" "User Commands" .SH NAME fail2ban-regex \- test Fail2ban "failregex" option .SH SYNOPSIS @@ -42,9 +42,15 @@ show program's version number and exit \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP +\fB\-c\fR CONFIG, \fB\-\-config\fR=\fI\,CONFIG\/\fR +set alternate config directory +.TP \fB\-d\fR DATEPATTERN, \fB\-\-datepattern\fR=\fI\,DATEPATTERN\/\fR set custom pattern used to match date/times .TP +\fB\-\-timezone\fR=\fI\,TIMEZONE\/\fR, \fB\-\-TZ\fR=\fI\,TIMEZONE\/\fR +set time\-zone used by convert time format +.TP \fB\-e\fR ENCODING, \fB\-\-encoding\fR=\fI\,ENCODING\/\fR File encoding. Default: system locale .TP @@ -57,7 +63,7 @@ DNS specified replacement of tags <HOST> in regexp addresses only) .TP \fB\-L\fR MAXLINES, \fB\-\-maxlines\fR=\fI\,MAXLINES\/\fR -maxlines for multi\-line regex +maxlines for multi\-line regex. .TP \fB\-m\fR JOURNALMATCH, \fB\-\-journalmatch\fR=\fI\,JOURNALMATCH\/\fR journalctl style matches overriding filter file. @@ -67,7 +73,13 @@ journalctl style matches overriding filter file. Log level for the Fail2Ban logger to use .TP \fB\-v\fR, \fB\-\-verbose\fR -Be verbose in output +Increase verbosity +.TP +\fB\-\-verbosity\fR=\fI\,VERBOSE\/\fR +Set numerical level of verbosity (0..4) +.TP +\fB\-\-verbose\-date\fR, \fB\-\-VD\fR +Verbose date patterns/regex in output .TP \fB\-D\fR, \fB\-\-debuggex\fR Produce debuggex.com urls for debugging there diff --git a/man/fail2ban-server.1 b/man/fail2ban-server.1 index 09dcd65a..711ecd6f 100644 --- a/man/fail2ban-server.1 +++ b/man/fail2ban-server.1 @@ -1,5 +1,5 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-SERVER "1" "September 2016" "fail2ban-server v0.10.0a2" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-SERVER "1" "August 2017" "fail2ban-server v0.10.0a2" "User Commands" .SH NAME fail2ban-server \- start the server .SH SYNOPSIS @@ -32,6 +32,9 @@ Note. If fail2ban running as systemd-service, for logging to the systemd-journal \fB\-d\fR dump configuration. For debugging .TP +\fB\-t\fR, \fB\-\-test\fR +test configuration (can be also specified with start parameters) +.TP \fB\-i\fR interactive mode .TP @@ -56,6 +59,9 @@ start server in async mode (for internal usage only, don't read configuration) \fB\-\-timeout\fR timeout to wait for the server (for internal usage only, don't read configuration) .TP +\fB\-\-str2sec\fR <STRING> +convert time abbreviation format to seconds +.TP \fB\-h\fR, \fB\-\-help\fR display this help message .TP diff --git a/man/fail2ban-testcases.1 b/man/fail2ban-testcases.1 index 26dbae96..0348d246 100644 --- a/man/fail2ban-testcases.1 +++ b/man/fail2ban-testcases.1 @@ -1,5 +1,5 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. -.TH FAIL2BAN-TESTCASES "1" "September 2016" "fail2ban-testcases 0.10.0a2" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. +.TH FAIL2BAN-TESTCASES "1" "August 2017" "fail2ban-testcases 0.10.0a2" "User Commands" .SH NAME fail2ban-testcases \- run Fail2Ban unit-tests .SH SYNOPSIS @@ -18,7 +18,10 @@ show this help message and exit \fB\-l\fR LOG_LEVEL, \fB\-\-log\-level\fR=\fI\,LOG_LEVEL\/\fR Log level for the logger to use during running tests .TP -\fB\-v\fR VERBOSITY, \fB\-\-verbosity\fR=\fI\,VERBOSITY\/\fR +\fB\-v\fR +Increase verbosity +.TP +\fB\-\-verbosity\fR=\fI\,VERBOSITY\/\fR Set numerical level of verbosity (0..4) .TP \fB\-\-log\-direct\fR diff --git a/man/jail.conf.5 b/man/jail.conf.5 index 2e333e5a..e939771b 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -90,11 +90,16 @@ indicates that the specified file is to be parsed after the current file. .RE Using Python "string interpolation" mechanisms, other definitions are allowed and can later be used within other definitions as %(name)s. -Additionally fail2ban has an extended interpolation feature named \fB%(known/parameter)s\fR (means last known option with name \fBparameter\fR). This interpolation makes possible to extend a stock filter or jail regexp in .local file (opposite to simply set failregex/ignoreregex that overwrites it), e.g. + +Fail2ban has more advanced syntax (similar python extended interpolation). This extended interpolation is using \fB%(section/parameter)s\fR to denote a value from a foreign section. +.br +Besides cross section interpolation the value of parameter in \fI[DEFAULT]\fR section can be retrieved with \fB%(default/parameter)s\fR. +.br +Fail2ban supports also another feature named \fB%(known/parameter)s\fR (means last known option with name \fBparameter\fR). This interpolation makes possible to extend a stock filter or jail regexp in .local file (opposite to simply set failregex/ignoreregex that overwrites it), e.g. .RS .nf -baduseragents = IE|wget +baduseragents = IE|wget|%(my-settings/baduseragents)s failregex = %(known/failregex)s useragent=%(baduseragents)s .fi @@ -178,6 +183,25 @@ Ensure syslog or the program that generates the log file isn't configured to com .B logencoding encoding of log files used for decoding. Default value of "auto" uses current system locale. .TP +.B logtimezone +Force the time zone for log lines that don't have one. + +If this option is not specified, log lines from which no explicit time zone has been found are interpreted by fail2ban in its own system time zone, and that may turn to be inappropriate. While the best practice is to configure the monitored applications to include explicit offsets, this option is meant to handle cases where that is not possible. + +The supported time zones in this option are those with fixed offset: Z, UTC[+-]hhmm (you can also use GMT as an alias to UTC). + +This option has no effect on log lines on which an explicit time zone has been found. +Examples: + +.RS +.nf + logtimezone = UTC + logtimezone = UTC+0200 + logtimezone = GMT-0100 +.fi +.RE + +.TP .B banaction banning action (default iptables-multiport) typically specified in the \fI[DEFAULT]\fR section for all jails. .br @@ -199,20 +223,23 @@ Arguments can be passed to actions to override the default values from the [Init Values can also be quoted (required when value includes a ","). More that one action can be specified (in separate lines). .RE .TP +.B ignoreself +boolean value (default true) indicates the banning of own IP addresses should be prevented +.TP .B ignoreip -list of IPs not to ban. They can include a CIDR mask too. +list of IPs not to ban. They can include a DNS resp. CIDR mask too. The option affects additionally to \fBignoreself\fR (if true) and don't need to contain own DNS resp. IPs of the running host. .TP .B ignorecommand -command that is executed to determine if the current candidate IP for banning should not be banned. +command that is executed to determine if the current candidate IP for banning (or failure-ID for raw IDs) should not be banned. The option affects additionally to \fBignoreself\fR and \fBignoreip\fR and will be first executed if both don't hit. .br IP will not be banned if command returns successfully (exit code 0). Like ACTION FILES, tags like <ip> are can be included in the ignorecommand value and will be substituted before execution. Currently only <ip> is supported however more will be added later. .TP .B bantime -effective ban duration (in seconds). +effective ban duration (in seconds or time abbreviation format). .TP .B findtime -time interval (in seconds) before the current time where failures will count towards a ban. +time interval (in seconds or time abbreviation format) before the current time where failures will count towards a ban. .TP .B maxretry number of failures that have to occur in the last \fBfindtime\fR seconds to ban then IP. @@ -256,6 +283,38 @@ action = smtp.py[dest=chris@example.com, actname=smtp-chris] smtp.py[dest=sally@example.com, actname=smtp-sally] .fi +.SH "TIME ABBREVIATION FORMAT" +The time entries in fail2ban configuration (like \fBfindtime\fR or \fBbantime\fR) can be provided as integer in seconds or as string using special abbreviation format (e. g. \fB600\fR is the same as \fB10m\fR). + +.TP +.B Abbreviation tokens: + +.RS +.nf +years?, yea?, yy? +months?, mon? +weeks?, wee?, ww? +days?, da, dd? +hours?, hou?, hh? +minutes?, min?, mm? +seconds?, sec?, ss? + +The question mark (?) means the optional character, so \fBday\fR as well as \fBdays\fR can be used. +.fi +.RE + +You can combine multiple tokens in format (separated with space resp. without separator), e. g.: \fB1y 6mo\fR or \fB1d12h30m\fR. +.br +Note that tokens \fBm\fR as well as \fBmm\fR means minutes, for month use abbreviation \fBmo\fR or \fBmon\fR. + +The time format can be tested using \fBfail2ban-client\fR: + +.RS +.nf +fail2ban-client --str2sec 1d12h +.fi +.RE + .SH "ACTION CONFIGURATION FILES (\fIaction.d/*.conf\fB)" Action files specify which commands are executed to ban and unban an IP address. |