diff options
93 files changed, 1631 insertions, 993 deletions
diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..0cbdbf83 --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +ChangeLog linguist-language=Markdown diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 00000000..543f316a --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,4 @@ +# These are supported funding model platforms + +github: [sebres] +custom: [paypal.me/sebres] diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index cb4b4bc6..00000000 --- a/.github/ISSUE_TEMPLATE.md +++ /dev/null @@ -1,49 +0,0 @@ -_We will be very grateful, if your problem was described as completely as possible, -enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident -within INFO mode), and configuration in particular of effected relevant settings -(e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular -jail troubleshooting). -Thank you in advance for the details, because such issues like "It does not work" -alone could not help to resolve anything! -Thanks! (remove this paragraph and other comments upon reading)_ - -### Environment: - -_Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, -and you can't verify that the issue persists in the recent release, better seek support -from the distribution you obtained Fail2Ban from_ - -- Fail2Ban version (including any possible distribution suffixes): -- OS, including release name/version: -- [ ] Fail2Ban installed via OS/distribution mechanisms -- [ ] You have not applied any additional foreign patches to the codebase -- [ ] Some customizations were done to the configuration (provide details below is so) - -### The issue: - -_Summary here_ - -#### Steps to reproduce - -#### Expected behavior - -#### Observed behavior - -#### Any additional information - -### Configuration, dump and another helpful excerpts - -#### Any customizations done to /etc/fail2ban/ configuration -``` -``` - -#### Relevant parts of /var/log/fail2ban.log file: -_preferably obtained while running fail2ban with `loglevel = 4`_ - -``` -``` - -#### Relevant lines from monitored log files in question: - -``` -```
\ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..33d94e10 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,70 @@ +--- +name: Bug report +about: Report a bug within the fail2ban engines (not filters or jails) +title: '[BR]: ' +labels: bug +assignees: '' + +--- + +<!-- + - Before reporting, please make sure to search the open and closed issues for any reports in the past. + - Use this issue template to report a bug in the fail2ban engine (not in a filter or jail). + - If you want to request a feature or a new filter, please use "Feature request" or "Filter request" instead. + - If you have rather some question, please open or join to some discussion. + + We will be very grateful, if your problem was described as completely as possible, + enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident + within INFO mode), and configuration in particular of effected relevant settings + (e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular + jail troubleshooting). + Thank you in advance for the details, because such issues like "It does not work" + alone could not help to resolve anything! + Thanks! + (you can remove this paragraph and other comments upon reading) +--> + +### Environment: + +<!-- + Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, + and you can't verify that the issue persists in the recent release, better seek support + from the distribution you obtained Fail2Ban from +--> + +- Fail2Ban version <!-- including any possible distribution suffixes --> : +- OS, including release name/version : +- [ ] Fail2Ban installed via OS/distribution mechanisms +- [ ] You have not applied any additional foreign patches to the codebase +- [ ] Some customizations were done to the configuration (provide details below is so) + +### The issue: + +<!-- summary here --> + +#### Steps to reproduce + +#### Expected behavior + +#### Observed behavior + +#### Any additional information + + +### Configuration, dump and another helpful excerpts + +#### Any customizations done to /etc/fail2ban/ configuration +<!-- put your configuration excerpts between next 2 lines --> +``` +``` + +#### Relevant parts of /var/log/fail2ban.log file: +<!-- preferably obtained while running fail2ban with `loglevel = 4` --> +<!-- put your log excerpt between next 2 lines --> +``` +``` + +#### Relevant lines from monitored log files: +<!-- put your log excerpt between next 2 lines --> +``` +``` diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..41812e82 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,35 @@ +--- +name: Feature request +about: Suggest an idea or an enhancement for this project +title: '[RFE]: ' +labels: enhancement +assignees: '' + +--- + +<!-- + - Before requesting, please make sure to search the open and closed issues for any requests in the past. + - Use this issue template to request a feature in the fail2ban engine (not a new filter or jail). + - If you want to request a new filter or failregex, please use "Filter request" instead. + - If you have rather some question, please open or join to some discussion. +--> + +#### Feature request type +<!-- + Please provide a summary description of the feature request. +--> + +#### Description +<!-- + Please describe the feature in more detail. +--> + +#### Considered alternatives +<!-- + A clear and concise description of any alternative solutions or features you've considered. +--> + +#### Any additional information +<!-- + Add any other context or screenshots about the feature request here. +--> diff --git a/.github/ISSUE_TEMPLATE/filter_request.md b/.github/ISSUE_TEMPLATE/filter_request.md new file mode 100644 index 00000000..caf02f90 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/filter_request.md @@ -0,0 +1,59 @@ +--- +name: Filter request +about: Request a new jail or filter to be supported or existing filter extended with new failregex +title: '[FR]: ' +labels: filter-request +assignees: '' + +--- + +<!-- + - Before requesting, please make sure to search the open and closed issues for any requests in the past. + - Sometimes failregex have been already requested before but are not implemented yet due to various reasons. + - If there are no hits for your concerns, please proceed otherwise add a comment to the related issue (also if it is closed). + - If you want to request a new feature, please use "Feature request" instead. + - If you have rather some question, please open or join to some discussion. +--> + +### Environment: + +<!-- + Fill out and check (`[x]`) the boxes which apply. +--> + +- Fail2Ban version <!-- including any possible distribution suffixes --> : +- OS, including release name/version : + +#### Service, project or product which log or journal should be monitored + +- Name of filter or jail in Fail2Ban (if already exists) : +- Service, project or product name, including release name/version : +- Repository or URL (if known) : +- Service type : +- Ports and protocols the service is listening : + +#### Log or journal information +<!-- Delete unrelated group --> + +<!-- Log file --> + +- Log file name(s) : + +<!-- Systemd journal --> + +- Journal identifier or unit name : + +#### Any additional information + + +### Relevant lines from monitored log files: + +#### failures in sense of fail2ban filter (fail2ban must match): +<!-- put your log excerpt between next 2 lines --> +``` +``` + +#### legitimate messages (fail2ban should not consider as failures): +<!-- put your log excerpt between next 2 lines --> +``` +``` diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 3a17ccc2..350d6ee2 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,7 +1,8 @@ Before submitting your PR, please review the following checklist: - [ ] **CHOOSE CORRECT BRANCH**: if filing a bugfix/enhancement - against 0.9.x series, choose `master` branch + against certain release version, choose `0.9`, `0.10` or `0.11` branch, + for dev-edition use `master` branch - [ ] **CONSIDER adding a unit test** if your PR resolves an issue - [ ] **LIST ISSUES** this PR resolves - [ ] **MAKE SURE** this PR doesn't break existing tests diff --git a/.travis.yml b/.travis.yml index 398c120a..502af5be 100644 --- a/.travis.yml +++ b/.travis.yml @@ -33,7 +33,8 @@ install: # coverage - travis_retry pip install coverage # coveralls (note coveralls doesn't support 2.6 now): - - if [[ $TRAVIS_PYTHON_VERSION != 2.6* ]]; then F2B_COV=1; else F2B_COV=0; fi + #- if [[ $TRAVIS_PYTHON_VERSION != 2.6* ]]; then F2B_COV=1; else F2B_COV=0; fi + - F2B_COV=1 - if [[ "$F2B_COV" = 1 ]]; then travis_retry pip install coveralls; fi # codecov: - travis_retry pip install codecov @@ -1,3 +1,4 @@ +<!-- vim: syntax=Markdown --> __ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ @@ -6,10 +7,50 @@ Fail2Ban: Changelog =================== +ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition +----------- + +### Compatibility +* the minimum supported python version is now 2.7, if you have previous python version + you can use the 0.11 version of fail2ban or upgrade python (or even build it from source). +* potential incompatibility by parsing of options of `backend`, `filter` and `action` parameters (if they + are partially incorrect), because fail2ban could throw an error now (doesn't silently bypass it anymore). +* to v.0.11: + - due to change of `actioncheck` behavior (gh-488), some actions can be incompatible as regards + the invariant check, if `actionban` or `actionunban` would not throw an error (exit code + different from 0) in case of unsane environment. + - actions that have used tag `<ip>` (instead of `<fid>` or `<F-ID>`) to get failure-ID may become + incompatible, if filter uses IP-related tags (like `<ADDR>` or `<HOST>`) additionally to `<F-ID>` + and the values are different (gh-3217) + +### Fixes +* readline fixed to consider interim new-line character as part of code point in multi-byte logs + (e. g. unicode encoding like utf-16be, utf-16le); +* `action.d/ufw.conf`: + - fixed handling on IPv6 (using prepend, gh-2331, gh-3018) + - application names containing spaces can be used now (gh-656, gh-1532, gh-3018) +* `filter.d/drupal-auth.conf` more strict regex, extended to match "Login attempt failed from" (gh-2742) + +### New Features and Enhancements +* `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair + of sane environment (in case of recognized unsane state) would only occur on action errors (e. g. + if ban or unban operations are exiting with other code as 0) +* better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file + (and hash calculation) +* file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion) +* actions differentiate tags `<ip>` and `<fid>` (`<F-ID>`), if IP-address deviates from ID then the value + of `<ip>` is not equal `<fid>` anymore (gh-3217) +* `action.d/ufw.conf` (gh-3018): + - new option `add` (default `prepend`), can be supplied as `insert 1` for ufw versions before v.0.36 (gh-2331, gh-3018) + - new options `kill-mode` and `kill` to drop established connections of intruder (see action for details, gh-3018) +* `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) + mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881 + + ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools ----------- -### Compatibility: +### Compatibility * to v.0.10: - 0.11 is totally compatible to 0.10 (configuration- and API-related stuff), but the database got some new tables and fields (auto-converted during the first start), so once updated to 0.11, you @@ -189,7 +230,7 @@ Yes, Hrrrm... ### New Features * new replacement tags for failregex to match subnets in form of IP-addresses with CIDR mask (gh-2559): - `<CIDR>` - helper regex to match CIDR (simple integer form of net-mask); - - `<SUBNET>` - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional); + - `<SUBNET>` - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional); * grouped tags (`<ADDR>`, `<HOST>`, `<SUBNET>`) recognize IP addresses enclosed in square brackets * new failregex-flag tag `<F-MLFGAINED>` for failregex, signaled that the access to service was gained (ATM used similar to tag `<F-NOFAIL>`, but it does not add the log-line to matches, gh-2279) @@ -278,6 +278,7 @@ to tune it. fail2ban-regex -D ... will present Debuggex URLs for the regexs and sample log files that you pass into it. In general use when using regex debuggers for generating fail2ban filters: + * use regex from the ./fail2ban-regex output (to ensure all substitutions are done) * replace <HOST> with (?&.ipv4) @@ -24,7 +24,6 @@ config/action.d/hostsdeny.conf config/action.d/ipfilter.conf config/action.d/ipfw.conf config/action.d/iptables-allports.conf -config/action.d/iptables-common.conf config/action.d/iptables.conf config/action.d/iptables-ipset-proto4.conf config/action.d/iptables-ipset-proto6-allports.conf @@ -2,7 +2,7 @@ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \ |_| \__,_|_|_/___|_.__/\__,_|_||_| - v0.11.0.dev1 20??/??/?? + v1.0.1.dev1 20??/??/?? ## Fail2Ban: ban hosts that cause multiple authentication errors @@ -33,7 +33,8 @@ Installation: this case, you should use that instead.** Required: -- [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) +- [Python2 >= 2.7 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) +- python-setuptools, python-distutils or python3-setuptools for installation from source Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify), may require: @@ -46,11 +47,11 @@ Optional: To install: - tar xvfj fail2ban-0.11.0.tar.bz2 - cd fail2ban-0.11.0 + tar xvfj fail2ban-1.0.1.tar.bz2 + cd fail2ban-1.0.1 sudo python setup.py install -Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, 0.11 +Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, master or 0.11 git clone https://github.com/fail2ban/fail2ban.git cd fail2ban @@ -89,11 +90,11 @@ fail2ban(1) and jail.conf(5) manpages for further references. Code status: ------------ -* travis-ci.org: [](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) +* travis-ci.org: [](https://travis-ci.org/fail2ban/fail2ban?branch=master) / [](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) -* coveralls.io: [](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch) +* coveralls.io: [](https://coveralls.io/github/fail2ban/fail2ban?branch=master) / [](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch) -* codecov.io: [](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch) +* codecov.io: [](https://codecov.io/gh/fail2ban/fail2ban/branch/master) / [](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch) Contact: -------- diff --git a/config/action.d/firewallcmd-ipset.conf b/config/action.d/firewallcmd-ipset.conf index 99541910..c36ba694 100644 --- a/config/action.d/firewallcmd-ipset.conf +++ b/config/action.d/firewallcmd-ipset.conf @@ -18,20 +18,45 @@ before = firewallcmd-common.conf [Definition] -actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt> +actionstart = <ipstype_<ipsettype>/actionstart> firewall-cmd --direct --add-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype> -actionflush = ipset flush <ipmset> +actionflush = <ipstype_<ipsettype>/actionflush> actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype> <actionflush> - ipset destroy <ipmset> + <ipstype_<ipsettype>/actionstop> -actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist +actionban = <ipstype_<ipsettype>/actionban> # actionprolong = %(actionban)s -actionunban = ipset del <ipmset> <ip> -exist +actionunban = <ipstype_<ipsettype>/actionunban> + +[ipstype_ipset] + +actionstart = ipset -exist create <ipmset> hash:ip timeout <default-ipsettime> <familyopt> + +actionflush = ipset flush <ipmset> + +actionstop = ipset destroy <ipmset> + +actionban = ipset -exist add <ipmset> <ip> timeout <ipsettime> + +actionunban = ipset -exist del <ipmset> <ip> + +[ipstype_firewalld] + +actionstart = firewall-cmd --direct --new-ipset=<ipmset> --type=hash:ip --option=timeout=<default-ipsettime> <firewalld_familyopt> + +# TODO: there doesn't seem to be an explicit way to invoke the ipset flush function using firewall-cmd +actionflush = + +actionstop = firewall-cmd --direct --delete-ipset=<ipmset> + +actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip> + +actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip> [Init] @@ -56,6 +81,12 @@ ipsettime = 0 # banaction = %(known/banaction)s[ipsettime='<timeout-bantime>'] timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0) +# Option: ipsettype +# Notes.: defines type of ipset used for match-set (firewalld or ipset) +# Values: firewalld or ipset +# Default: ipset +ipsettype = ipset + # Option: actiontype # Notes.: defines additions to the blocking rule # Values: leave empty to block all attempts from the host @@ -75,14 +106,16 @@ multiport = -p <protocol> -m multiport --dports <port> ipmset = f2b-<name> familyopt = +firewalld_familyopt = [Init?family=inet6] ipmset = f2b-<name>6 familyopt = family inet6 +firewalld_familyopt = --option=family=inet6 # DEV NOTES: # -# Author: Edgar Hoch and Daniel Black +# Author: Edgar Hoch, Daniel Black, Sergey Brester and Mihail Politaev # firewallcmd-new / iptables-ipset-proto6 combined for maximium goodness diff --git a/config/action.d/iptables-allports.conf b/config/action.d/iptables-allports.conf index caf9ab81..51c4694d 100644 --- a/config/action.d/iptables-allports.conf +++ b/config/action.d/iptables-allports.conf @@ -4,52 +4,12 @@ # Modified: Yaroslav O. Halchenko <debian@onerussian.com> # made active on all ports from original iptables.conf # -# +# Obsolete: superseded by iptables[type=allports] [INCLUDES] -before = iptables-common.conf - +before = iptables.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = <iptables> -N f2b-<name> - <iptables> -A f2b-<name> -j <returntype> - <iptables> -I <chain> -p <protocol> -j f2b-<name> - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = <iptables> -D <chain> -p <protocol> -j f2b-<name> - <actionflush> - <iptables> -X f2b-<name> - -# Option: actioncheck -# Notes.: command executed once before each actionban command -# Values: CMD -# -actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]' - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype> - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype> - -[Init] - +type = allports diff --git a/config/action.d/iptables-common.conf b/config/action.d/iptables-common.conf deleted file mode 100644 index e016ef2f..00000000 --- a/config/action.d/iptables-common.conf +++ /dev/null @@ -1,92 +0,0 @@ -# Fail2Ban configuration file -# -# Author: Daniel Black -# -# This is a included configuration file and includes the definitions for the iptables -# used in all iptables based actions by default. -# -# The user can override the defaults in iptables-common.local -# -# Modified: Alexander Koeppe <format_c@online.de>, Serg G. Brester <serg.brester@sebres.de> -# made config file IPv6 capable (see new section Init?family=inet6) - -[INCLUDES] - -after = iptables-blocktype.local - iptables-common.local -# iptables-blocktype.local is obsolete - -[Definition] - -# Option: actionflush -# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) -# Values: CMD -# -actionflush = <iptables> -F f2b-<name> - - -[Init] - -# Option: chain -# Notes specifies the iptables chain to which the Fail2Ban rules should be -# added -# Values: STRING Default: INPUT -chain = INPUT - -# Default name of the chain -# -name = default - -# Option: port -# Notes.: specifies port to monitor -# Values: [ NUM | STRING ] Default: -# -port = ssh - -# Option: protocol -# Notes.: internally used by config reader for interpolations. -# Values: [ tcp | udp | icmp | all ] Default: tcp -# -protocol = tcp - -# Option: blocktype -# Note: This is what the action does with rules. This can be any jump target -# as per the iptables man page (section 8). Common values are DROP -# REJECT, REJECT --reject-with icmp-port-unreachable -# Values: STRING -blocktype = REJECT --reject-with icmp-port-unreachable - -# Option: returntype -# Note: This is the default rule on "actionstart". This should be RETURN -# in all (blocking) actions, except REJECT in allowing actions. -# Values: STRING -returntype = RETURN - -# Option: lockingopt -# Notes.: Option was introduced to iptables to prevent multiple instances from -# running concurrently and causing irratic behavior. -w was introduced -# in iptables 1.4.20, so might be absent on older systems -# See https://github.com/fail2ban/fail2ban/issues/1122 -# Values: STRING -lockingopt = -w - -# Option: iptables -# Notes.: Actual command to be executed, including common to all calls options -# Values: STRING -iptables = iptables <lockingopt> - - -[Init?family=inet6] - -# Option: blocktype (ipv6) -# Note: This is what the action does with rules. This can be any jump target -# as per the iptables man page (section 8). Common values are DROP -# REJECT, REJECT --reject-with icmp6-port-unreachable -# Values: STRING -blocktype = REJECT --reject-with icmp6-port-unreachable - -# Option: iptables (ipv6) -# Notes.: Actual command to be executed, including common to all calls options -# Values: STRING -iptables = ip6tables <lockingopt> - diff --git a/config/action.d/iptables-ipset-proto4.conf b/config/action.d/iptables-ipset-proto4.conf index 99ebbf8c..37624284 100644 --- a/config/action.d/iptables-ipset-proto4.conf +++ b/config/action.d/iptables-ipset-proto4.conf @@ -19,7 +19,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] @@ -28,7 +28,7 @@ before = iptables-common.conf # Values: CMD # actionstart = ipset --create f2b-<name> iphash - <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype> + <_ipt_add_rules> # Option: actionflush @@ -41,7 +41,7 @@ actionflush = ipset --flush f2b-<name> # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) # Values: CMD # -actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set f2b-<name> src -j <blocktype> +actionstop = <_ipt_del_rules> <actionflush> ipset --destroy f2b-<name> @@ -61,5 +61,6 @@ actionban = ipset --test f2b-<name> <ip> || ipset --add f2b-<name> <ip> # actionunban = ipset --test f2b-<name> <ip> && ipset --del f2b-<name> <ip> -[Init] +# Several capabilities used internaly: +rule-jump = -m set --match-set f2b-<name> src -j <blocktype> diff --git a/config/action.d/iptables-ipset-proto6-allports.conf b/config/action.d/iptables-ipset-proto6-allports.conf index 67d7947b..1aa7fd6f 100644 --- a/config/action.d/iptables-ipset-proto6-allports.conf +++ b/config/action.d/iptables-ipset-proto6-allports.conf @@ -15,73 +15,13 @@ # # Modified: Alexander Koeppe <format_c@online.de>, Serg G. Brester <serg.brester@sebres.de> # made config file IPv6 capable (see new section Init?family=inet6) +# +# Obsolete: superseded by iptables-ipset[type=allports] [INCLUDES] -before = iptables-common.conf +before = iptables-ipset.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt> - <iptables> -I <chain> -m set --match-set <ipmset> src -j <blocktype> - -# Option: actionflush -# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) -# Values: CMD -# -actionflush = ipset flush <ipmset> - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = <iptables> -D <chain> -m set --match-set <ipmset> src -j <blocktype> - <actionflush> - ipset destroy <ipmset> - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist - -# actionprolong = %(actionban)s - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = ipset del <ipmset> <ip> -exist - -[Init] - -# Option: default-ipsettime -# Notes: specifies default timeout in seconds (handled default ipset timeout only) -# Values: [ NUM ] Default: 0 (no timeout, managed by fail2ban by unban) -default-ipsettime = 0 - -# Option: ipsettime -# Notes: specifies ticket timeout (handled ipset timeout only) -# Values: [ NUM ] Default: 0 (managed by fail2ban by unban) -ipsettime = 0 - -# expresion to caclulate timeout from bantime, example: -# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>'] -timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0) - -ipmset = f2b-<name> -familyopt = - - -[Init?family=inet6] - -ipmset = f2b-<name>6 -familyopt = family inet6 +type = allports diff --git a/config/action.d/iptables-ipset-proto6.conf b/config/action.d/iptables-ipset-proto6.conf index 87601027..ef744984 100644 --- a/config/action.d/iptables-ipset-proto6.conf +++ b/config/action.d/iptables-ipset-proto6.conf @@ -15,73 +15,13 @@ # # Modified: Alexander Koeppe <format_c@online.de>, Serg G. Brester <serg.brester@sebres.de> # made config file IPv6 capable (see new section Init?family=inet6) +# +# Obsolete: superseded by iptables-ipset[type=multiport] [INCLUDES] -before = iptables-common.conf +before = iptables-ipset.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt> - <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype> - -# Option: actionflush -# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) -# Values: CMD -# -actionflush = ipset flush <ipmset> - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype> - <actionflush> - ipset destroy <ipmset> - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist - -# actionprolong = %(actionban)s - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = ipset del <ipmset> <ip> -exist - -[Init] - -# Option: default-ipsettime -# Notes: specifies default timeout in seconds (handled default ipset timeout only) -# Values: [ NUM ] Default: 0 (no timeout, managed by fail2ban by unban) -default-ipsettime = 0 - -# Option: ipsettime -# Notes: specifies ticket timeout (handled ipset timeout only) -# Values: [ NUM ] Default: 0 (managed by fail2ban by unban) -ipsettime = 0 - -# expresion to caclulate timeout from bantime, example: -# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>'] -timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0) - -ipmset = f2b-<name> -familyopt = - - -[Init?family=inet6] - -ipmset = f2b-<name>6 -familyopt = family inet6 +type = multiport diff --git a/config/action.d/iptables-ipset.conf b/config/action.d/iptables-ipset.conf new file mode 100644 index 00000000..b44e6ec4 --- /dev/null +++ b/config/action.d/iptables-ipset.conf @@ -0,0 +1,90 @@ +# Fail2Ban configuration file +# +# Authors: Sergey G Brester (sebres), Daniel Black, Alexander Koeppe +# +# This is for ipset protocol 6 (and hopefully later) (ipset v6.14). +# Use ipset -V to see the protocol and version. Version 4 should use +# iptables-ipset-proto4.conf. +# +# This requires the program ipset which is normally in package called ipset. +# +# IPset was a feature introduced in the linux kernel 2.6.39 and 3.0.0 kernels. +# +# If you are running on an older kernel you make need to patch in external +# modules. +# + +[INCLUDES] + +before = iptables.conf + +[Definition] + +# Option: actionstart +# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). +# Values: CMD +# +actionstart = ipset -exist create <ipmset> hash:ip timeout <default-ipsettime> <familyopt> + <_ipt_add_rules> + +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = ipset flush <ipmset> + +# Option: actionstop +# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) +# Values: CMD +# +actionstop = <_ipt_del_rules> + <actionflush> + ipset destroy <ipmset> + +# Option: actionban +# Notes.: command executed when banning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionban = ipset -exist add <ipmset> <ip> timeout <ipsettime> + +# actionprolong = %(actionban)s + +# Option: actionunban +# Notes.: command executed when unbanning an IP. Take care that the +# command is executed with Fail2Ban user rights. +# Tags: See jail.conf(5) man page +# Values: CMD +# +actionunban = ipset -exist del <ipmset> <ip> + +# Several capabilities used internaly: + +rule-jump = -m set --match-set <ipmset> src -j <blocktype> + + +[Init] + +# Option: default-ipsettime +# Notes: specifies default timeout in seconds (handled default ipset timeout only) +# Values: [ NUM ] Default: 0 (no timeout, managed by fail2ban by unban) +default-ipsettime = 0 + +# Option: ipsettime +# Notes: specifies ticket timeout (handled ipset timeout only) +# Values: [ NUM ] Default: 0 (managed by fail2ban by unban) +ipsettime = 0 + +# expresion to caclulate timeout from bantime, example: +# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>'] +timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0) + +ipmset = f2b-<name> +familyopt = + + +[Init?family=inet6] + +ipmset = f2b-<name>6 +familyopt = family inet6 diff --git a/config/action.d/iptables-multiport-log.conf b/config/action.d/iptables-multiport-log.conf index df126dbf..322a7491 100644 --- a/config/action.d/iptables-multiport-log.conf +++ b/config/action.d/iptables-multiport-log.conf @@ -11,7 +11,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] diff --git a/config/action.d/iptables-multiport.conf b/config/action.d/iptables-multiport.conf index 41b00c54..008208e0 100644 --- a/config/action.d/iptables-multiport.conf +++ b/config/action.d/iptables-multiport.conf @@ -3,50 +3,12 @@ # Author: Cyril Jaquier # Modified by Yaroslav Halchenko for multiport banning # +# Obsolete: superseded by iptables[type=multiport] [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = <iptables> -N f2b-<name> - <iptables> -A f2b-<name> -j <returntype> - <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name> - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name> - <actionflush> - <iptables> -X f2b-<name> - -# Option: actioncheck -# Notes.: command executed once before each actionban command -# Values: CMD -# -actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]' - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype> - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype> - -[Init] - +type = multiport diff --git a/config/action.d/iptables-new.conf b/config/action.d/iptables-new.conf index 39a17099..170cb934 100644 --- a/config/action.d/iptables-new.conf +++ b/config/action.d/iptables-new.conf @@ -4,51 +4,12 @@ # Copied from iptables.conf and modified by Yaroslav Halchenko # to fulfill the needs of bugreporter dbts#350746. # -# +# Obsolete: superseded by iptables[pre-rule='-m state --state NEW<sp>'] [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] -# Option: actionstart -# Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). -# Values: CMD -# -actionstart = <iptables> -N f2b-<name> - <iptables> -A f2b-<name> -j <returntype> - <iptables> -I <chain> -m state --state NEW -p <protocol> --dport <port> -j f2b-<name> - -# Option: actionstop -# Notes.: command executed at the stop of jail (or at the end of Fail2Ban) -# Values: CMD -# -actionstop = <iptables> -D <chain> -m state --state NEW -p <protocol> --dport <port> -j f2b-<name> - <actionflush> - <iptables> -X f2b-<name> - -# Option: actioncheck -# Notes.: command executed once before each actionban command -# Values: CMD -# -actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]' - -# Option: actionban -# Notes.: command executed when banning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype> - -# Option: actionunban -# Notes.: command executed when unbanning an IP. Take care that the -# command is executed with Fail2Ban user rights. -# Tags: See jail.conf(5) man page -# Values: CMD -# -actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype> - -[Init] - +pre-rule = -m state --state NEW<sp>
\ No newline at end of file diff --git a/config/action.d/iptables-xt_recent-echo.conf b/config/action.d/iptables-xt_recent-echo.conf index 97449222..c3c175b3 100644 --- a/config/action.d/iptables-xt_recent-echo.conf +++ b/config/action.d/iptables-xt_recent-echo.conf @@ -7,10 +7,14 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] +_ipt_chain_rule = -m recent --update --seconds 3600 --name <iptname> -j <blocktype> +_ipt_for_proto-iter = +_ipt_for_proto-done = + # Option: actionstart # Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). # Values: CMD @@ -33,7 +37,9 @@ before = iptables-common.conf # own rules. The 3600 second timeout is independent and acts as a # safeguard in case the fail2ban process dies unexpectedly. The # shorter of the two timeouts actually matters. -actionstart = if [ `id -u` -eq 0 ];then <iptables> -I <chain> -m recent --update --seconds 3600 --name <iptname> -j <blocktype>;fi +actionstart = if [ `id -u` -eq 0 ];then + { %(_ipt_check_rule)s >/dev/null 2>&1; } || { <iptables> -I <chain> %(_ipt_chain_rule)s; } + fi # Option: actionflush # @@ -46,13 +52,15 @@ actionflush = # Values: CMD # actionstop = echo / > /proc/net/xt_recent/<iptname> - if [ `id -u` -eq 0 ];then <iptables> -D <chain> -m recent --update --seconds 3600 --name <iptname> -j <blocktype>;fi + if [ `id -u` -eq 0 ];then + <iptables> -D <chain> %(_ipt_chain_rule)s; + fi # Option: actioncheck -# Notes.: command executed once before each actionban command +# Notes.: command executed as invariant check (error by ban) # Values: CMD # -actioncheck = test -e /proc/net/xt_recent/<iptname> +actioncheck = { <iptables> -C <chain> %(_ipt_chain_rule)s; } && test -e /proc/net/xt_recent/<iptname> # Option: actionban # Notes.: command executed when banning an IP. Take care that the @@ -72,7 +80,7 @@ actionunban = echo -<ip> > /proc/net/xt_recent/<iptname> [Init] -iptname = f2b-<name> +iptname = f2b-<name> [Init?family=inet6] diff --git a/config/action.d/iptables.conf b/config/action.d/iptables.conf index 8ed5fdad..67d496f5 100644 --- a/config/action.d/iptables.conf +++ b/config/action.d/iptables.conf @@ -1,28 +1,35 @@ # Fail2Ban configuration file # -# Author: Cyril Jaquier -# +# Authors: Sergey G. Brester (sebres), Cyril Jaquier, Daniel Black, +# Yaroslav O. Halchenko, Alexander Koeppe et al. # -[INCLUDES] +[Definition] -before = iptables-common.conf +# Option: type +# Notes.: type of the action. +# Values: [ oneport | multiport | allports ] Default: oneport +# +type = oneport -[Definition] +# Option: actionflush +# Notes.: command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action) +# Values: CMD +# +actionflush = <iptables> -F f2b-<name> # Option: actionstart # Notes.: command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false). # Values: CMD # -actionstart = <iptables> -N f2b-<name> - <iptables> -A f2b-<name> -j <returntype> - <iptables> -I <chain> -p <protocol> --dport <port> -j f2b-<name> +actionstart = { <iptables> -C f2b-<name> -j <returntype> >/dev/null 2>&1; } || { <iptables> -N f2b-<name> || true; <iptables> -A f2b-<name> -j <returntype>; } + <_ipt_add_rules> # Option: actionstop # Notes.: command executed at the stop of jail (or at the end of Fail2Ban) # Values: CMD # -actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name> +actionstop = <_ipt_del_rules> <actionflush> <iptables> -X f2b-<name> @@ -30,7 +37,7 @@ actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name> # Notes.: command executed once before each actionban command # Values: CMD # -actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]' +actioncheck = <_ipt_check_rules> # Option: actionban # Notes.: command executed when banning an IP. Take care that the @@ -48,5 +55,108 @@ actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype> # actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype> +# Option: pre-rule +# Notes.: prefix parameter(s) inserted to the begin of rule. No default (empty) +# +pre-rule = + +rule-jump = -j <_ipt_rule_target> + +# Several capabilities used internaly: + +_ipt_for_proto-iter = for proto in $(echo '<protocol>' | sed 's/,/ /g'); do +_ipt_for_proto-done = done + +_ipt_add_rules = <_ipt_for_proto-iter> + { %(_ipt_check_rule)s >/dev/null 2>&1; } || { <iptables> -I <chain> %(_ipt_chain_rule)s; } + <_ipt_for_proto-done> + +_ipt_del_rules = <_ipt_for_proto-iter> + <iptables> -D <chain> %(_ipt_chain_rule)s + <_ipt_for_proto-done> + +_ipt_check_rules = <_ipt_for_proto-iter> + %(_ipt_check_rule)s + <_ipt_for_proto-done> + +_ipt_chain_rule = <pre-rule><ipt_<type>/_chain_rule> +_ipt_check_rule = <iptables> -C <chain> %(_ipt_chain_rule)s +_ipt_rule_target = f2b-<name> + +[ipt_oneport] + +_chain_rule = -p $proto --dport <port> <rule-jump> + +[ipt_multiport] + +_chain_rule = -p $proto -m multiport --dports <port> <rule-jump> + +[ipt_allports] + +_chain_rule = -p $proto <rule-jump> + + [Init] +# Option: chain +# Notes specifies the iptables chain to which the Fail2Ban rules should be +# added +# Values: STRING Default: INPUT +chain = INPUT + +# Default name of the chain +# +name = default + +# Option: port +# Notes.: specifies port to monitor +# Values: [ NUM | STRING ] Default: +# +port = ssh + +# Option: protocol +# Notes.: internally used by config reader for interpolations. +# Values: [ tcp | udp | icmp | all ] Default: tcp +# +protocol = tcp + +# Option: blocktype +# Note: This is what the action does with rules. This can be any jump target +# as per the iptables man page (section 8). Common values are DROP +# REJECT, REJECT --reject-with icmp-port-unreachable +# Values: STRING +blocktype = REJECT --reject-with icmp-port-unreachable + +# Option: returntype +# Note: This is the default rule on "actionstart". This should be RETURN +# in all (blocking) actions, except REJECT in allowing actions. +# Values: STRING +returntype = RETURN + +# Option: lockingopt +# Notes.: Option was introduced to iptables to prevent multiple instances from +# running concurrently and causing irratic behavior. -w was introduced +# in iptables 1.4.20, so might be absent on older systems +# See https://github.com/fail2ban/fail2ban/issues/1122 +# Values: STRING +lockingopt = -w + +# Option: iptables +# Notes.: Actual command to be executed, including common to all calls options +# Values: STRING +iptables = iptables <lockingopt> + + +[Init?family=inet6] + +# Option: blocktype (ipv6) +# Note: This is what the action does with rules. This can be any jump target +# as per the iptables man page (section 8). Common values are DROP +# REJECT, REJECT --reject-with icmp6-port-unreachable +# Values: STRING +blocktype = REJECT --reject-with icmp6-port-unreachable + +# Option: iptables (ipv6) +# Notes.: Actual command to be executed, including common to all calls options +# Values: STRING +iptables = ip6tables <lockingopt> diff --git a/config/action.d/symbiosis-blacklist-allports.conf b/config/action.d/symbiosis-blacklist-allports.conf index 6fb7d0af..7208b293 100644 --- a/config/action.d/symbiosis-blacklist-allports.conf +++ b/config/action.d/symbiosis-blacklist-allports.conf @@ -5,7 +5,7 @@ [INCLUDES] -before = iptables-common.conf +before = iptables.conf [Definition] @@ -41,6 +41,11 @@ actionban = echo 'all' >| /etc/symbiosis/firewall/blacklist.d/<ip>.auto actionunban = rm -f /etc/symbiosis/firewall/blacklist.d/<ip>.auto <iptables> -D <chain> -s <ip> -j <blocktype> || : +# [TODO] Flushing is currently not implemented for symbiosis blacklist.d +# +actionflush = + + [Init] # Option: chain diff --git a/config/action.d/ufw.conf b/config/action.d/ufw.conf index d2f731f2..c9ff7f37 100644 --- a/config/action.d/ufw.conf +++ b/config/action.d/ufw.conf @@ -13,16 +13,45 @@ actionstop = actioncheck = -actionban = [ -n "<application>" ] && app="app <application>" - ufw insert <insertpos> <blocktype> from <ip> to <destination> $app +# ufw does "quickly process packets for which we already have a connection" in before.rules, +# therefore all related sockets should be closed +# actionban is using `ss` to do so, this only handles IPv4 and IPv6. -actionunban = [ -n "<application>" ] && app="app <application>" - ufw delete <blocktype> from <ip> to <destination> $app +actionban = if [ -n "<application>" ] && ufw app info "<application>" + then + ufw <add> <blocktype> from <ip> to <destination> app "<application>" comment "<comment>" + else + ufw <add> <blocktype> from <ip> to <destination> comment "<comment>" + fi + <kill> + +actionunban = if [ -n "<application>" ] && ufw app info "<application>" + then + ufw delete <blocktype> from <ip> to <destination> app "<application>" + else + ufw delete <blocktype> from <ip> to <destination> + fi + +# Option: kill-mode +# Notes.: can be set to ss or conntrack (may be extended later with other modes) to immediately drop all connections from banned IP, default empty (no kill) +# Example: banaction = ufw[kill-mode=ss] +kill-mode = + +# intern conditional parameter used to provide killing mode after ban: +_kill_ = +_kill_ss = ss -K dst "[<ip>]" +_kill_conntrack = conntrack -D -s "<ip>" + +# Option: kill +# Notes.: can be used to specify custom killing feature, by default depending on option kill-mode +# Examples: banaction = ufw[kill='ss -K "( sport = :http || sport = :https )" dst "[<ip>]"'] +# banaction = ufw[kill='cutter "<ip>"'] +kill = <_kill_<kill-mode>> [Init] -# Option: insertpos -# Notes.: The position number in the firewall list to insert the block rule -insertpos = 1 +# Option: add +# Notes.: can be set to "insert 1" to insert a rule at certain position (here 1): +add = prepend # Option: blocktype # Notes.: reject or deny @@ -36,6 +65,10 @@ destination = any # Notes.: application from sudo ufw app list application = +# Option: comment +# Notes.: comment for rule added by fail2ban +comment = by Fail2Ban after <failures> attempts against <name> + # DEV NOTES: # # Author: Guilhem Lettron diff --git a/config/filter.d/apache-fakegooglebot.conf b/config/filter.d/apache-fakegooglebot.conf index 729410ad..ee23656a 100644 --- a/config/filter.d/apache-fakegooglebot.conf +++ b/config/filter.d/apache-fakegooglebot.conf @@ -2,11 +2,11 @@ [Definition] -failregex = ^<HOST> .*Googlebot.*$ +failregex = ^\s*<HOST> \S+ \S+(?: \S+)?\s+\S+ "[A-Z]+ /\S* [^"]*" \d+ \d+ \"[^"]*\" "[^"]*\bGooglebot/[^"]*" ignoreregex = -datepattern = ^[^\[]*\[({DATE}) +datepattern = ^[^\[]*(\[{DATE}\s*\]) {^LN-BEG} # DEV Notes: diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 68472495..e15d7bfe 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -21,7 +21,7 @@ log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])?:? [^:]+ prefregex = ^%(__prefix_line)s%(log_prefix)s <F-CONTENT>.+</F-CONTENT>$ failregex = ^Registration from '[^']*' failed for '<HOST>(:\d+)?' - (?:Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$ - ^Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context + ^Call from '[^']*' \((?:(?:TCP|UDP):)?<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context ^(?:Host )?<HOST> (?:failed (?:to authenticate\b|MD5 authentication\b)|tried to authenticate with nonexistent user\b) ^No registration for peer '[^']*' \(from <HOST>\)$ ^hacking attempt detected '<HOST>'$ diff --git a/config/filter.d/drupal-auth.conf b/config/filter.d/drupal-auth.conf index b60abe3e..2404cc6d 100644 --- a/config/filter.d/drupal-auth.conf +++ b/config/filter.d/drupal-auth.conf @@ -14,7 +14,7 @@ before = common.conf [Definition] -failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|<HOST>\|.+\|.+\|\d\|.*\|Login attempt failed for .+\.$ +failregex = ^%(__prefix_line)s(?:https?:\/\/)[^|]+\|[^|]+\|[^|]+\|<ADDR>\|(?:[^|]*\|)*Login attempt failed (?:for|from) <F-USER>[^|]+</F-USER>\.$ ignoreregex = diff --git a/config/filter.d/monitorix.conf b/config/filter.d/monitorix.conf new file mode 100644 index 00000000..ff69f1bc --- /dev/null +++ b/config/filter.d/monitorix.conf @@ -0,0 +1,25 @@ +# Fail2Ban filter for Monitorix (HTTP built-in server) +# + +[INCLUDES] + +before = common.conf + +[Definition] + +_daemon = monitorix-httpd + +# Option: failregex +# Notes.: regex to match the password failures messages in the logfile. The +# host must be matched by a group named "host". The tag "<HOST>" can +# be used for standard IP/hostname matching and is only an alias for +# (?:::f{4,6}:)?(?P<host>\S+) +# Values: TEXT +# +failregex = ^(?:\s+-)?\s*(?:NOTEXIST|AUTHERR|NOTALLOWED) - <ADDR>\b + +# Option: ignoreregex +# Notes.: regex to ignore. If this regex matches, the line is ignored. +# Values: TEXT +# +ignoreregex = diff --git a/config/filter.d/mssql-auth.conf b/config/filter.d/mssql-auth.conf new file mode 100644 index 00000000..65bbd917 --- /dev/null +++ b/config/filter.d/mssql-auth.conf @@ -0,0 +1,15 @@ +# Fail2Ban filter for failed MSSQL Server authentication attempts + +[Definition] + +failregex = ^\s*Logon\s+Login failed for user '<F-USER>(?:[^']*|.*)</F-USER>'\. [^'\[]+\[CLIENT: <ADDR>\]$ + + +# DEV Notes: +# Tested with SQL Server 2019 on Ubuntu 18.04 +# +# Example: +# 2020-02-24 14:48:55.12 Logon Login failed for user 'root'. Reason: Could not find a login matching the name provided. [CLIENT: 127.0.0.1] +# +# Author: Rüdiger Olschewsky +#
\ No newline at end of file diff --git a/config/filter.d/named-refused.conf b/config/filter.d/named-refused.conf index 8a0b1b8c..6dbbbf81 100644 --- a/config/filter.d/named-refused.conf +++ b/config/filter.d/named-refused.conf @@ -22,7 +22,7 @@ [Definition] # Daemon name -_daemon=named +_daemon=named(?:-\w+)? # Shortcuts for easier comprehension of the failregex diff --git a/config/filter.d/nginx-bad-request.conf b/config/filter.d/nginx-bad-request.conf new file mode 100644 index 00000000..12c14ab7 --- /dev/null +++ b/config/filter.d/nginx-bad-request.conf @@ -0,0 +1,16 @@ +# Fail2Ban filter to match bad requests to nginx +# + +[Definition] + +# The request often doesn't contain a method, only some encoded garbage +# This will also match requests that are entirely empty +failregex = ^<HOST> - \S+ \[\] "[^"]*" 400 + +datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? + ^[^\[]*\[({DATE}) + {^LN-BEG} + +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + +# Author: Jan Przybylak diff --git a/config/filter.d/nginx-botsearch.conf b/config/filter.d/nginx-botsearch.conf index 0be895b2..2bd23072 100644 --- a/config/filter.d/nginx-botsearch.conf +++ b/config/filter.d/nginx-botsearch.conf @@ -17,7 +17,9 @@ datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,] ^[^\[]*\[({DATE}) {^LN-BEG} +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + # DEV Notes: # Based on apache-botsearch filter # -# Author: Frantisek Sumsal
\ No newline at end of file +# Author: Frantisek Sumsal diff --git a/config/filter.d/nginx-http-auth.conf b/config/filter.d/nginx-http-auth.conf index 93341cd2..71806e85 100644 --- a/config/filter.d/nginx-http-auth.conf +++ b/config/filter.d/nginx-http-auth.conf @@ -3,15 +3,32 @@ [Definition] +mode = normal -failregex = ^ \[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$ +mdre-auth = ^\s*\[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$ +mdre-fallback = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: <HOST> + +mdre-normal = %(mdre-auth)s +mdre-aggressive = %(mdre-auth)s + %(mdre-fallback)s + +failregex = <mdre-<mode>> ignoreregex = datepattern = {^LN-BEG} +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + # DEV NOTES: +# mdre-auth: # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files # Extensive search of all nginx auth failures not done yet. # # Author: Daniel Black + +# mdre-fallback: +# Ban people checking for TLS_FALLBACK_SCSV repeatedly +# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608 +# Author: Stephan Orlowsky + diff --git a/config/filter.d/nginx-limit-req.conf b/config/filter.d/nginx-limit-req.conf index e23548ab..2f45e831 100644 --- a/config/filter.d/nginx-limit-req.conf +++ b/config/filter.d/nginx-limit-req.conf @@ -44,3 +44,6 @@ failregex = ^\s*\[[a-z]+\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by ignoreregex = datepattern = {^LN-BEG} + +journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx + diff --git a/config/filter.d/nsd.conf b/config/filter.d/nsd.conf index bfd99544..0589c16c 100644 --- a/config/filter.d/nsd.conf +++ b/config/filter.d/nsd.conf @@ -22,10 +22,10 @@ _daemon = nsd # (?:::f{4,6}:)?(?P<host>[\w\-.^_]+) # Values: TEXT -failregex = ^%(__prefix_line)sinfo: ratelimit block .* query <HOST> TYPE255$ - ^%(__prefix_line)sinfo: .* <HOST> refused, no acl matches\.$ +failregex = ^%(__prefix_line)sinfo: ratelimit block .* query <ADDR> TYPE255$ + ^%(__prefix_line)sinfo: .* from(?: client)? <ADDR> refused, no acl matches\.?$ ignoreregex = datepattern = {^LN-BEG}Epoch - {^LN-BEG}
\ No newline at end of file + {^LN-BEG} diff --git a/config/filter.d/postfix.conf b/config/filter.d/postfix.conf index 01d8cb0b..b374f472 100644 --- a/config/filter.d/postfix.conf +++ b/config/filter.d/postfix.conf @@ -16,8 +16,11 @@ _pref = [A-Z]{4} prefregex = ^%(__prefix_line)s<mdpr-<mode>> <F-CONTENT>.+</F-CONTENT>$ +# Extended RE for normal mode to match reject by unknown users or undeliverable address, can be set to empty to avoid this: +exre-user = |[Uu](?:ser unknown|ndeliverable address) + mdpr-normal = (?:\w+: (?:milter-)?reject:|(?:improper command pipelining|too many errors) after \S+) -mdre-normal=^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|User unknown|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match)\b +mdre-normal=^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match%(exre-user)s)\b ^from [^[]*\[<HOST>\]%(_port)s:? mdpr-auth = warning: @@ -33,7 +36,9 @@ mdre-rbl = ^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45]54 [45]\.7\.1 Service mdpr-more = %(mdpr-normal)s mdre-more = %(mdre-normal)s -mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))) +# Includes some of the log messages described in +# <http://www.postfix.org/POSTSCREEN_README.html>. +mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+|COMMAND (?:TIME|COUNT|LENGTH) LIMIT) mdre-ddos = ^from [^[]*\[<HOST>\]%(_port)s:? mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s) diff --git a/config/filter.d/scanlogd.conf b/config/filter.d/scanlogd.conf new file mode 100644 index 00000000..d3fe78b0 --- /dev/null +++ b/config/filter.d/scanlogd.conf @@ -0,0 +1,17 @@ +# Fail2Ban filter for port scans detected by scanlogd + +[INCLUDES] + +# Read common prefixes. If any customizations available -- read them from +# common.local +before = common.conf + +[Definition] + +_daemon = scanlogd + +failregex = ^%(__prefix_line)s<ADDR>(?::<F-PORT/>)? to \S+ ports\b + +ignoreregex = + +# Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> diff --git a/config/filter.d/zoneminder.conf b/config/filter.d/zoneminder.conf index cc82755a..8e8ed432 100644 --- a/config/filter.d/zoneminder.conf +++ b/config/filter.d/zoneminder.conf @@ -5,17 +5,23 @@ before = apache-common.conf [Definition] -# pattern: [Wed Apr 27 23:12:07.736196 2016] [:error] [pid 2460] [client 10.1.1.1:47296] WAR [Login denied for user "test"], referer: https://zoneminderurl/index.php -# +# patterns: [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ +# [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/ +# [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/ # # Option: failregex -# Notes.: regex to match the password failure messages in the logfile. +# Notes.: regex to match the login failure and non-existent user error messages in the logfile. + +prefregex = ^%(_apache_error_client)s (?:ERR|WAR) <F-CONTENT>\[(?:Login denied|Could not retrieve).*</F-CONTENT>$ -failregex = ^%(_apache_error_client)s WAR \[Login denied for user "[^"]*"\] +failregex = ^\[Login denied for user "<F-USER>[^"]*</F-USER>"\] + ^\[Could not retrieve user <F-USER>\S*</F-USER> ignoreregex = # Notes: -# Tested on Zoneminder 1.29.0 +# Tested on Zoneminder 1.29 and 1.35.21 +# +# Zoneminder versions > 1.3x use "ERR" and < 1.3x use "WAR" level logs, so i've kept both for compatibility reasons # # Author: John Marzella diff --git a/config/jail.conf b/config/jail.conf index a4f67896..fe8db527 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -67,7 +67,7 @@ before = paths-debian.conf # more aggressive example of formula has the same values only for factor "2.0 / 2.885385" : #bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor) -# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding +# "bantime.multipliers" used to calculate next value of ban time instead of formula, corresponding # previously ban count and given "bantime.factor" (for multipliers default is 1); # following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count, # always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours @@ -77,7 +77,7 @@ before = paths-debian.conf #bantime.multipliers = 1 5 30 60 300 720 1440 2880 # "bantime.overalljails" (if true) specifies the search of IP in the database will be executed -# cross over all jails, if false (dafault), only current jail of the ban IP will be searched +# cross over all jails, if false (default), only current jail of the ban IP will be searched #bantime.overalljails = false # -------------------- @@ -346,7 +346,7 @@ maxretry = 2 port = http,https logpath = %(apache_access_log)s maxretry = 1 -ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip> +ignorecommand = %(fail2ban_confpath)s/filter.d/ignorecommands/apache-fakegooglebot <ip> [apache-modsecurity] @@ -370,8 +370,11 @@ banaction = %(banaction_allports)s logpath = /opt/openhab/logs/request.log +# To use more aggressive http-auth modes set filter parameter "mode" in jail.local: +# normal (default), aggressive (combines all), auth or fallback +# See "tests/files/logs/nginx-http-auth" or "filter.d/nginx-http-auth.conf" for usage example and details. [nginx-http-auth] - +# mode = normal port = http,https logpath = %(nginx_error_log)s @@ -387,8 +390,10 @@ logpath = %(nginx_error_log)s port = http,https logpath = %(nginx_error_log)s -maxretry = 2 +[nginx-bad-request] +port = http,https +logpath = %(nginx_access_log)s # Ban attackers that try to use PHP's URL-fopen() functionality # through GET/POST variables. - Experimental, with more than a year @@ -792,6 +797,14 @@ logpath = %(mysql_log)s backend = %(mysql_backend)s +[mssql-auth] +# Default configuration for Microsoft SQL Server for Linux +# See the 'mssql-conf' manpage how to change logpath or port +logpath = /var/opt/mssql/log/errorlog +port = 1433 +filter = mssql-auth + + # Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf') [mongodb-auth] # change port when running with "--shardsvr" or "--configsvr" runtime operation @@ -957,3 +970,11 @@ logpath = %(apache_error_log)s # see `filter.d/traefik-auth.conf` for details and service example. port = http,https logpath = /var/log/traefik/access.log + +[scanlogd] +logpath = %(syslog_local0)s +banaction = %(banaction_allports)s + +[monitorix] +port = 8080 +logpath = /var/log/monitorix-httpd diff --git a/config/paths-common.conf b/config/paths-common.conf index 7383cafe..4f6a5f71 100644 --- a/config/paths-common.conf +++ b/config/paths-common.conf @@ -91,6 +91,3 @@ mysql_log = %(syslog_daemon)s mysql_backend = %(default_backend)s roundcube_errors_log = /var/log/roundcube/errors - -# Directory with ignorecommand scripts -ignorecommands_dir = /etc/fail2ban/filter.d/ignorecommands diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py index 6ea18fda..a7053034 100755 --- a/fail2ban/client/fail2banclient.py +++ b/fail2ban/client/fail2banclient.py @@ -230,7 +230,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): logSys.log(5, ' client phase %s', phase) if not stream: return False - # wait a litle bit for phase "start-ready" before enter active waiting: + # wait a little bit for phase "start-ready" before enter active waiting: if phase is not None: Utils.wait_for(lambda: phase.get('start-ready', None) is not None, 0.5, 0.001) phase['configure'] = (True if stream else False) diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index de7cde60..b1795588 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -289,9 +289,6 @@ class Fail2banRegex(object): def output(self, line): if not self._opts.out: output(line) - def decode_line(self, line): - return FileContainer.decode_line('<LOG>', self._encoding, line) - def encode_line(self, line): return line.encode(self._encoding, 'ignore') @@ -523,10 +520,14 @@ class Fail2banRegex(object): def _prepaireOutput(self): """Prepares output- and fetch-function corresponding given '--out' option (format)""" ofmt = self._opts.out - if ofmt in ('id', 'ip'): + if ofmt in ('id', 'fid'): def _out(ret): for r in ret: output(r[1]) + elif ofmt == 'ip': + def _out(ret): + for r in ret: + output(r[3].get('ip', r[1])) elif ofmt == 'msg': def _out(ret): for r in ret: @@ -723,10 +724,6 @@ class Fail2banRegex(object): return True - def file_lines_gen(self, hdlr): - for line in hdlr: - yield self.decode_line(line) - def start(self, args): cmd_log, cmd_regex = args[:2] @@ -745,10 +742,10 @@ class Fail2banRegex(object): if os.path.isfile(cmd_log): try: - hdlr = open(cmd_log, 'rb') + test_lines = FileContainer(cmd_log, self._encoding, doOpen=True) + self.output( "Use log file : %s" % cmd_log ) self.output( "Use encoding : %s" % self._encoding ) - test_lines = self.file_lines_gen(hdlr) except IOError as e: # pragma: no cover output( e ) return False diff --git a/fail2ban/client/filterreader.py b/fail2ban/client/filterreader.py index 413f125e..24341014 100644 --- a/fail2ban/client/filterreader.py +++ b/fail2ban/client/filterreader.py @@ -72,8 +72,9 @@ class FilterReader(DefinitionInitConfigReader): def _fillStream(stream, opts, jailName): prio0idx = 0 for opt, value in opts.iteritems(): + # Do not send a command if the value is not set (empty). + if value is None: continue if opt in ("failregex", "ignoreregex"): - if value is None: continue multi = [] for regex in value.split('\n'): # Do not send a command if the rule is empty. @@ -91,8 +92,6 @@ class FilterReader(DefinitionInitConfigReader): elif opt in ('datepattern'): stream.append(["set", jailName, opt, value]) elif opt == 'journalmatch': - # Do not send a command if the match is empty. - if value is None: continue for match in value.split("\n"): if match == '': continue stream.append( diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index f3ccf7db..37746d4c 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -121,9 +121,12 @@ class JailReader(ConfigReader): def getOptions(self): + basedir = self.getBaseDir() + # Before interpolation (substitution) add static options always available as default: self.merge_defaults({ - "fail2ban_version": version + "fail2ban_version": version, + "fail2ban_confpath": basedir }) try: @@ -146,7 +149,7 @@ class JailReader(ConfigReader): raise JailDefError("Invalid filter definition %r: %s" % (flt, e)) self.__filter = FilterReader( filterName, self.__name, filterOpt, - share_config=self.share_config, basedir=self.getBaseDir()) + share_config=self.share_config, basedir=basedir) ret = self.__filter.read() if not ret: raise JailDefError("Unable to read the filter %r" % filterName) @@ -186,13 +189,13 @@ class JailReader(ConfigReader): "addaction", actOpt.pop("actname", os.path.splitext(actName)[0]), os.path.join( - self.getBaseDir(), "action.d", actName), + basedir, "action.d", actName), json.dumps(actOpt), ]) else: action = ActionReader( actName, self.__name, actOpt, - share_config=self.share_config, basedir=self.getBaseDir()) + share_config=self.share_config, basedir=basedir) ret = action.read() if ret: action.getOptions(self.__opts) diff --git a/fail2ban/protocol.py b/fail2ban/protocol.py index 0a4c84ed..58102b55 100644 --- a/fail2ban/protocol.py +++ b/fail2ban/protocol.py @@ -134,7 +134,7 @@ protocol = [ ["get <JAIL> ignoreregex", "gets the list of regular expressions which matches patterns to ignore for <JAIL>"], ["get <JAIL> findtime", "gets the time for which the filter will look back for failures for <JAIL>"], ["get <JAIL> bantime", "gets the time a host is banned for <JAIL>"], -["get <JAIL> datepattern", "gets the patern used to match date/times for <JAIL>"], +["get <JAIL> datepattern", "gets the pattern used to match date/times for <JAIL>"], ["get <JAIL> usedns", "gets the usedns setting for <JAIL>"], ["get <JAIL> banip [<SEP>|--with-time]", "gets the list of of banned IP addresses for <JAIL>. Optionally the separator character ('<SEP>', default is space) or the option '--with-time' (printing the times of ban) may be specified. The IPs are ordered by end of ban."], ["get <JAIL> maxretry", "gets the number of failures allowed for <JAIL>"], diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index 99ab2250..16ff6621 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -410,7 +410,7 @@ class CommandAction(ActionBase): cmd = self.replaceTag(tag, self._properties, conditional=('family='+family if family else ''), cache=self.__substCache) - if '<' not in cmd or not family: return cmd + if not family or '<' not in cmd: return cmd # replace family as dynamic tags, important - don't cache, no recursion and auto-escape here: cmd = self.replaceDynamicTags(cmd, {'family':family}) return cmd @@ -977,31 +977,38 @@ class CommandAction(ActionBase): except (KeyError, TypeError): family = '' - # invariant check: - if self.actioncheck: - # don't repair/restore if unban (no matter): - def _beforeRepair(): - if cmd == '<actionunban>' and not self._properties.get('actionrepair_on_unban'): - self._logSys.error("Invariant check failed. Unban is impossible.") - return False - return True - # check and repair if broken: - ret = self._invariantCheck(family, _beforeRepair, forceStart=(cmd != '<actionunban>')) - # if not sane (and not restored) return: - if ret != 1: - return False - - # Replace static fields - realCmd = self.replaceTag(cmd, self._properties, - conditional=('family='+family if family else ''), cache=self.__substCache) + repcnt = 0 + while True: - # Replace dynamical tags, important - don't cache, no recursion and auto-escape here - if aInfo is not None: - realCmd = self.replaceDynamicTags(realCmd, aInfo) - else: - realCmd = cmd + # got some error, do invariant check: + if repcnt and self.actioncheck: + # don't repair/restore if unban (no matter): + def _beforeRepair(): + if cmd == '<actionunban>' and not self._properties.get('actionrepair_on_unban'): + self._logSys.error("Invariant check failed. Unban is impossible.") + return False + return True + # check and repair if broken: + ret = self._invariantCheck(family, _beforeRepair, forceStart=(cmd != '<actionunban>')) + # if not sane (and not restored) return: + if ret != 1: + return False - return self.executeCmd(realCmd, self.timeout) + # Replace static fields + realCmd = self.replaceTag(cmd, self._properties, + conditional=('family='+family if family else ''), cache=self.__substCache) + + # Replace dynamical tags, important - don't cache, no recursion and auto-escape here + if aInfo is not None: + realCmd = self.replaceDynamicTags(realCmd, aInfo) + else: + realCmd = cmd + + # try execute command: + ret = self.executeCmd(realCmd, self.timeout) + repcnt += 1 + if ret or repcnt > 1: + return ret @staticmethod def executeCmd(realCmd, timeout=60, **kwargs): diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index 83c137ae..7470c01d 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -32,10 +32,7 @@ try: from collections.abc import Mapping except ImportError: from collections import Mapping -try: - from collections import OrderedDict -except ImportError: - OrderedDict = dict +from collections import OrderedDict from .banmanager import BanManager, BanTicket from .ipdns import IPAddr @@ -84,7 +81,7 @@ class Actions(JailThread, Mapping): self._jail = jail self._actions = OrderedDict() ## The ban manager. - self.__banManager = BanManager() + self.banManager = BanManager() self.banEpoch = 0 self.__lastConsistencyCheckTM = 0 ## Precedence of ban (over unban), so max number of tickets banned (to call an unban check): @@ -203,7 +200,7 @@ class Actions(JailThread, Mapping): def setBanTime(self, value): value = MyTime.str2seconds(value) - self.__banManager.setBanTime(value) + self.banManager.setBanTime(value) logSys.info(" banTime: %s" % value) ## @@ -212,10 +209,10 @@ class Actions(JailThread, Mapping): # @return the time def getBanTime(self): - return self.__banManager.getBanTime() + return self.banManager.getBanTime() def getBanned(self, ids): - lst = self.__banManager.getBanList() + lst = self.banManager.getBanList() if not ids: return lst if len(ids) == 1: @@ -230,7 +227,7 @@ class Actions(JailThread, Mapping): list The list of banned IP addresses. """ - return self.__banManager.getBanList(ordered=True, withTime=withTime) + return self.banManager.getBanList(ordered=True, withTime=withTime) def addBannedIP(self, ip): """Ban an IP or list of IPs.""" @@ -282,7 +279,7 @@ class Actions(JailThread, Mapping): if db and self._jail.database is not None: self._jail.database.delBan(self._jail, ip) # Find the ticket with the IP. - ticket = self.__banManager.getTicketByID(ip) + ticket = self.banManager.getTicketByID(ip) if ticket is not None: # Unban the IP. self.__unBan(ticket) @@ -291,7 +288,7 @@ class Actions(JailThread, Mapping): if not isinstance(ip, IPAddr): ipa = IPAddr(ip) if not ipa.isSingle: # subnet (mask/cidr) or raw (may be dns/hostname): - ips = filter(ipa.contains, self.__banManager.getBanList()) + ips = filter(ipa.contains, self.banManager.getBanList()) if ips: return self.removeBannedIP(ips, db, ifexists) # not found: @@ -350,7 +347,7 @@ class Actions(JailThread, Mapping): continue # wait for ban (stop if gets inactive, pending ban or unban): bancnt = 0 - wt = min(self.sleeptime, self.__banManager._nextUnbanTime - MyTime.time()) + wt = min(self.sleeptime, self.banManager._nextUnbanTime - MyTime.time()) logSys.log(5, "Actions: wait for pending tickets %s (default %s)", wt, self.sleeptime) if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, wt): bancnt = self.__checkBan() @@ -397,7 +394,12 @@ class Actions(JailThread, Mapping): "ipfailures": lambda self: self._mi4ip(True).getAttempt(), "ipjailfailures": lambda self: self._mi4ip().getAttempt(), # raw ticket info: - "raw-ticket": lambda self: repr(self.__ticket) + "raw-ticket": lambda self: repr(self.__ticket), + # jail info: + "jail.banned": lambda self: self.__jail.actions.banManager.size(), + "jail.banned_total": lambda self: self.__jail.actions.banManager.getBanTotal(), + "jail.found": lambda self: self.__jail.filter.failManager.size(), + "jail.found_total": lambda self: self.__jail.filter.failManager.getFailTotal() } __slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip') @@ -494,11 +496,11 @@ class Actions(JailThread, Mapping): for ticket in tickets: bTicket = BanTicket.wrap(ticket) - btime = ticket.getBanTime(self.__banManager.getBanTime()) - ip = bTicket.getIP() + btime = ticket.getBanTime(self.banManager.getBanTime()) + ip = bTicket.getID() aInfo = self._getActionInfo(bTicket) reason = {} - if self.__banManager.addBanTicket(bTicket, reason=reason): + if self.banManager.addBanTicket(bTicket, reason=reason): cnt += 1 # report ticket to observer, to check time should be increased and hereafter observer writes ban to database (asynchronous) if Observers.Main is not None and not bTicket.restored: @@ -558,7 +560,7 @@ class Actions(JailThread, Mapping): # and increase ticket time if "bantime.increment" set) if cnt: logSys.debug("Banned %s / %s, %s ticket(s) in %r", cnt, - self.__banManager.getBanTotal(), self.__banManager.size(), self._jail.name) + self.banManager.getBanTotal(), self.banManager.size(), self._jail.name) return cnt def __reBan(self, ticket, actions=None, log=True): @@ -573,10 +575,10 @@ class Actions(JailThread, Mapping): Ticket to reban """ actions = actions or self._actions - ip = ticket.getIP() + ip = ticket.getID() aInfo = self._getActionInfo(ticket) if log: - logSys.notice("[%s] Reban %s%s", self._jail.name, aInfo["ip"], (', action %r' % actions.keys()[0] if len(actions) == 1 else '')) + logSys.notice("[%s] Reban %s%s", self._jail.name, ip, (', action %r' % actions.keys()[0] if len(actions) == 1 else '')) for name, action in actions.iteritems(): try: logSys.debug("[%s] action %r: reban %s", self._jail.name, name, ip) @@ -598,7 +600,7 @@ class Actions(JailThread, Mapping): def _prolongBan(self, ticket): # prevent to prolong ticket that was removed in-between, # if it in ban list - ban time already prolonged (and it stays there): - if not self.__banManager._inBanList(ticket): return + if not self.banManager._inBanList(ticket): return # do actions : aInfo = None for name, action in self._actions.iteritems(): @@ -623,13 +625,13 @@ class Actions(JailThread, Mapping): Unban IP addresses which are outdated. """ - lst = self.__banManager.unBanList(MyTime.time(), maxCount) + lst = self.banManager.unBanList(MyTime.time(), maxCount) for ticket in lst: self.__unBan(ticket) cnt = len(lst) if cnt: logSys.debug("Unbanned %s, %s ticket(s) in %r", - cnt, self.__banManager.size(), self._jail.name) + cnt, self.banManager.size(), self._jail.name) return cnt def __flushBan(self, db=False, actions=None, stop=False): @@ -643,10 +645,10 @@ class Actions(JailThread, Mapping): log = True if actions is None: logSys.debug(" Flush ban list") - lst = self.__banManager.flushBanList() + lst = self.banManager.flushBanList() else: log = False # don't log "[jail] Unban ..." if removing actions only. - lst = iter(self.__banManager) + lst = iter(self.banManager) cnt = 0 # first we'll execute flush for actions supporting this operation: unbactions = {} @@ -683,7 +685,7 @@ class Actions(JailThread, Mapping): self.__unBan(ticket, actions=actions, log=log) cnt += 1 logSys.debug(" Unbanned %s, %s ticket(s) in %r", - cnt, self.__banManager.size(), self._jail.name) + cnt, self.banManager.size(), self._jail.name) return cnt def __unBan(self, ticket, actions=None, log=True): @@ -701,10 +703,10 @@ class Actions(JailThread, Mapping): unbactions = self._actions else: unbactions = actions - ip = ticket.getIP() + ip = ticket.getID() aInfo = self._getActionInfo(ticket) if log: - logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"]) + logSys.notice("[%s] Unban %s", self._jail.name, ip) for name, action in unbactions.iteritems(): try: logSys.debug("[%s] action %r: unban %s", self._jail.name, name, ip) @@ -726,18 +728,18 @@ class Actions(JailThread, Mapping): logSys.warning("Unsupported extended jail status flavor %r. Supported: %s" % (flavor, supported_flavors)) # Always print this information (basic) if flavor != "short": - banned = self.__banManager.getBanList() + banned = self.banManager.getBanList() cnt = len(banned) else: - cnt = self.__banManager.size() + cnt = self.banManager.size() ret = [("Currently banned", cnt), - ("Total banned", self.__banManager.getBanTotal())] + ("Total banned", self.banManager.getBanTotal())] if flavor != "short": ret += [("Banned IP list", banned)] if flavor == "cymru": - cymru_info = self.__banManager.getBanListExtendedCymruInfo() + cymru_info = self.banManager.getBanListExtendedCymruInfo() ret += \ - [("Banned ASN list", self.__banManager.geBanListExtendedASN(cymru_info)), - ("Banned Country list", self.__banManager.geBanListExtendedCountry(cymru_info)), - ("Banned RIR list", self.__banManager.geBanListExtendedRIR(cymru_info))] + [("Banned ASN list", self.banManager.geBanListExtendedASN(cymru_info)), + ("Banned Country list", self.banManager.geBanListExtendedCountry(cymru_info)), + ("Banned RIR list", self.banManager.geBanListExtendedRIR(cymru_info))] return ret diff --git a/fail2ban/server/database.py b/fail2ban/server/database.py index ed736a7a..8a7abb3c 100644 --- a/fail2ban/server/database.py +++ b/fail2ban/server/database.py @@ -502,7 +502,7 @@ class Fail2BanDb(object): except TypeError: firstLineMD5 = None - if not firstLineMD5 and (pos or md5): + if firstLineMD5 is None and (pos or md5 is not None): cur.execute( "INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) " "VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos)) @@ -599,7 +599,7 @@ class Fail2BanDb(object): ticket : BanTicket Ticket of the ban to be added. """ - ip = str(ticket.getIP()) + ip = str(ticket.getID()) try: del self._bansMergedCache[(ip, jail)] except KeyError: diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 3657ea48..68968284 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -553,7 +553,7 @@ class Filter(JailThread): ticket = None if isinstance(ip, FailTicket): ticket = ip - ip = ticket.getIP() + ip = ticket.getID() elif not isinstance(ip, IPAddr): ip = IPAddr(ip) return self._inIgnoreIPList(ip, ticket, log_ignore) @@ -702,10 +702,7 @@ class Filter(JailThread): """Processes the line for failures and populates failManager """ try: - for element in self.processLine(line, date): - ip = element[1] - unixTime = element[2] - fail = element[3] + for (_, ip, unixTime, fail) in self.processLine(line, date): logSys.debug("Processing line with time:%s and ip:%s", unixTime, ip) # ensure the time is not in the future, e. g. by some estimated (assumed) time: @@ -724,7 +721,7 @@ class Filter(JailThread): self.performBan(ip) # report to observer - failure was found, for possibly increasing of it retry counter (asynchronous) if Observers.Main is not None: - Observers.Main.add('failureFound', self.failManager, self.jail, tick) + Observers.Main.add('failureFound', self.jail, tick) self.procLines += 1 # every 100 lines check need to perform service tasks: if self.procLines % 100 == 0: @@ -843,11 +840,9 @@ class Filter(JailThread): failList = list() ll = logSys.getEffectiveLevel() - returnRawHost = self.returnRawHost - cidr = IPAddr.CIDR_UNSPEC - if self.__useDns == "raw": - returnRawHost = True - cidr = IPAddr.CIDR_RAW + defcidr = IPAddr.CIDR_UNSPEC + if self.__useDns == "raw" or self.returnRawHost: + defcidr = IPAddr.CIDR_RAW if self.__lineBufferSize > 1: self.__lineBuffer.append(tupleLine) @@ -910,7 +905,8 @@ class Filter(JailThread): if not self.checkAllRegex or self.__lineBufferSize > 1: self.__lineBuffer, buf = failRegex.getUnmatchedTupleLines(), None # merge data if multi-line failure: - raw = returnRawHost + cidr = defcidr + raw = (defcidr == IPAddr.CIDR_RAW) if preGroups: currFail, fail = fail, preGroups.copy() fail.update(currFail) @@ -929,49 +925,50 @@ class Filter(JailThread): # failure-id: fid = fail.get('fid') # ip-address or host: - host = fail.get('ip4') - if host is not None: + ip = fail.get('ip4') + if ip is not None: cidr = int(fail.get('cidr') or IPAddr.FAM_IPv4) raw = True else: - host = fail.get('ip6') - if host is not None: + ip = fail.get('ip6') + if ip is not None: cidr = int(fail.get('cidr') or IPAddr.FAM_IPv6) raw = True - if host is None: - host = fail.get('dns') - if host is None: - # first try to check we have mlfid case (cache connection id): - if fid is None and mlfid is None: - # if no failure-id also (obscure case, wrong regex), throw error inside getFailID: - fid = failRegex.getFailID() - host = fid - cidr = IPAddr.CIDR_RAW - raw = True + else: + ip = fail.get('dns') + if ip is None: + # first try to check we have mlfid case (cache connection id): + if fid is None and mlfid is None: + # if no failure-id also (obscure case, wrong regex), throw error inside getFailID: + fid = failRegex.getFailID() + ip = fid + raw = True # if mlfid case (not failure): - if host is None: + if ip is None: if ll <= 7: logSys.log(7, "No failure-id by mlfid %r in regex %s: %s", mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier")) fail['mlfpending'] = 1; # mark failure is pending if not self.checkAllRegex and self.ignorePending: return failList - ips = [None] + fids = [None] # if raw - add single ip or failure-id, # otherwise expand host to multiple ips using dns (or ignore it if not valid): elif raw: - ip = IPAddr(host, cidr) - # check host equal failure-id, if not - failure with complex id: - if fid is not None and fid != host: - ip = IPAddr(fid, IPAddr.CIDR_RAW) - ips = [ip] + # check ip/host equal failure-id, if not - failure with complex id: + if fid is None or fid == ip: + fid = IPAddr(ip, cidr) + else: + fail['ip'] = IPAddr(ip, cidr) + fid = IPAddr(fid, defcidr) + fids = [fid] # otherwise, try to use dns conversion: else: - ips = DNSUtils.textToIp(host, self.__useDns) + fids = DNSUtils.textToIp(ip, self.__useDns) # if checkAllRegex we must make a copy (to be sure next RE doesn't change merged/cached failure): if self.checkAllRegex and mlfid is not None: fail = fail.copy() # append failure with match to the list: - for ip in ips: - failList.append([failRegexIndex, ip, date, fail]) + for fid in fids: + failList.append([failRegexIndex, fid, date, fail]) if not self.checkAllRegex: break except RegexException as e: # pragma: no cover - unsure if reachable @@ -1142,14 +1139,14 @@ class FileFilter(Filter): while not self.idle: line = log.readline() if not self.active: break; # jail has been stopped - if not line: + if line is None: # The jail reached the bottom, simply set in operation for this log # (since we are first time at end of file, growing is only possible after modifications): log.inOperation = True break # acquire in operation from log and process: self.inOperation = inOperation if inOperation is not None else log.inOperation - self.processLineAndAdd(line.rstrip('\r\n')) + self.processLineAndAdd(line) finally: log.close() if self.jail.database is not None: @@ -1172,6 +1169,8 @@ class FileFilter(Filter): if logSys.getEffectiveLevel() <= logging.DEBUG: logSys.debug("Seek to find time %s (%s), file size %s", date, MyTime.time2str(date), fs) + if not fs: + return minp = container.getPos() maxp = fs tryPos = minp @@ -1195,8 +1194,8 @@ class FileFilter(Filter): dateTimeMatch = None nextp = None while True: - line = container.readline() - if not line: + line = container.readline(False) + if line is None: break (timeMatch, template) = self.dateDetector.matchTime(line) if timeMatch: @@ -1314,25 +1313,34 @@ except ImportError: # pragma: no cover class FileContainer: - def __init__(self, filename, encoding, tail=False): + def __init__(self, filename, encoding, tail=False, doOpen=False): self.__filename = filename + self.waitForLineEnd = True self.setEncoding(encoding) self.__tail = tail self.__handler = None + self.__pos = 0 + self.__pos4hash = 0 + self.__hash = '' + self.__hashNextTime = time.time() + 30 # Try to open the file. Raises an exception if an error occurred. handler = open(filename, 'rb') - stats = os.fstat(handler.fileno()) - self.__ino = stats.st_ino + if doOpen: # fail2ban-regex only (don't need to reopen it and check for rotation) + self.__handler = handler + return try: - firstLine = handler.readline() - # Computes the MD5 of the first line. - self.__hash = md5sum(firstLine).hexdigest() - # Start at the beginning of file if tail mode is off. - if tail: - handler.seek(0, 2) - self.__pos = handler.tell() - else: - self.__pos = 0 + stats = os.fstat(handler.fileno()) + self.__ino = stats.st_ino + if stats.st_size: + firstLine = handler.readline() + # first line available and contains new-line: + if firstLine != firstLine.rstrip(b'\r\n'): + # Computes the MD5 of the first line. + self.__hash = md5sum(firstLine).hexdigest() + # if tail mode scroll to the end of file + if tail: + handler.seek(0, 2) + self.__pos = handler.tell() finally: handler.close() ## shows that log is in operation mode (expecting new messages only from here): @@ -1351,6 +1359,10 @@ class FileContainer: return self.__filename def getFileSize(self): + h = self.__handler + if h is not None: + stats = os.fstat(h.fileno()) + return stats.st_size return os.path.getsize(self.__filename); def setEncoding(self, encoding): @@ -1369,38 +1381,54 @@ class FileContainer: def setPos(self, value): self.__pos = value - def open(self): - self.__handler = open(self.__filename, 'rb') - # Set the file descriptor to be FD_CLOEXEC - fd = self.__handler.fileno() - flags = fcntl.fcntl(fd, fcntl.F_GETFD) - fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) - # Stat the file before even attempting to read it - stats = os.fstat(self.__handler.fileno()) - if not stats.st_size: - # yoh: so it is still an empty file -- nothing should be - # read from it yet - # print "D: no content -- return" - return False - firstLine = self.__handler.readline() - # Computes the MD5 of the first line. - myHash = md5sum(firstLine).hexdigest() - ## print "D: fn=%s hashes=%s/%s inos=%s/%s pos=%s rotate=%s" % ( - ## self.__filename, self.__hash, myHash, stats.st_ino, self.__ino, self.__pos, - ## self.__hash != myHash or self.__ino != stats.st_ino) - ## sys.stdout.flush() - # Compare hash and inode - if self.__hash != myHash or self.__ino != stats.st_ino: - logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename) - self.__hash = myHash - self.__ino = stats.st_ino - self.__pos = 0 - # Sets the file pointer to the last position. - self.__handler.seek(self.__pos) + def open(self, forcePos=None): + h = open(self.__filename, 'rb') + try: + # Set the file descriptor to be FD_CLOEXEC + fd = h.fileno() + flags = fcntl.fcntl(fd, fcntl.F_GETFD) + fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) + myHash = self.__hash + # Stat the file before even attempting to read it + stats = os.fstat(h.fileno()) + rotflg = stats.st_size < self.__pos or stats.st_ino != self.__ino + if rotflg or not len(myHash) or time.time() > self.__hashNextTime: + myHash = '' + firstLine = h.readline() + # Computes the MD5 of the first line (if it is complete) + if firstLine != firstLine.rstrip(b'\r\n'): + myHash = md5sum(firstLine).hexdigest() + self.__hashNextTime = time.time() + 30 + elif stats.st_size == self.__pos: + myHash = self.__hash + # Compare size, hash and inode + if rotflg or myHash != self.__hash: + if self.__hash != '': + logSys.log(logging.MSG, "Log rotation detected for %s, reason: %r", self.__filename, + (stats.st_size, self.__pos, stats.st_ino, self.__ino, myHash, self.__hash)) + self.__ino = stats.st_ino + self.__pos = 0 + self.__hash = myHash + # if nothing to read from file yet (empty or no new data): + if forcePos is not None: + self.__pos = forcePos + elif stats.st_size <= self.__pos: + return False + # Sets the file pointer to the last position. + h.seek(self.__pos) + # leave file open (to read content): + self.__handler = h; h = None + finally: + # close (no content or error only) + if h: + h.close(); h = None return True def seek(self, offs, endLine=True): h = self.__handler + if h is None: + self.open(offs) + h = self.__handler # seek to given position h.seek(offs, 0) # goto end of next line @@ -1418,6 +1446,9 @@ class FileContainer: try: return line.decode(enc, 'strict') except (UnicodeDecodeError, UnicodeEncodeError) as e: + # avoid warning if got incomplete end of line (e. g. '\n' in "...[0A" followed by "00]..." for utf-16le: + if (e.end == len(line) and line[e.start] in b'\r\n'): + return line[0:e.start].decode(enc, 'replace') global _decode_line_warn lev = 7 if not _decode_line_warn.get(filename, 0): @@ -1426,29 +1457,85 @@ class FileContainer: logSys.log(lev, "Error decoding line from '%s' with '%s'.", filename, enc) if logSys.getEffectiveLevel() <= lev: - logSys.log(lev, "Consider setting logencoding=utf-8 (or another appropriate" - " encoding) for this jail. Continuing" - " to process line ignoring invalid characters: %r", + logSys.log(lev, + "Consider setting logencoding to appropriate encoding for this jail. " + "Continuing to process line ignoring invalid characters: %r", line) # decode with replacing error chars: line = line.decode(enc, 'replace') return line - def readline(self): + def readline(self, complete=True): + """Read line from file + + In opposite to pythons readline it doesn't return new-line, + so returns either the line if line is complete (and complete=True) or None + if line is not complete (and complete=True) or there is no content to read. + If line is complete (and complete is True), it also shift current known + position to begin of next line. + + Also it is safe against interim new-line bytes (e. g. part of multi-byte char) + in given encoding. + """ if self.__handler is None: return "" - return FileContainer.decode_line( - self.getFileName(), self.getEncoding(), self.__handler.readline()) + # read raw bytes up to \n char: + b = self.__handler.readline() + if not b: + return None + bl = len(b) + # convert to log-encoding (new-line char could disappear if it is part of multi-byte sequence): + r = FileContainer.decode_line( + self.getFileName(), self.getEncoding(), b) + # trim new-line at end and check the line was written complete (contains a new-line): + l = r.rstrip('\r\n') + if complete: + if l == r: + # try to fill buffer in order to find line-end in log encoding: + fnd = 0 + while 1: + r = self.__handler.readline() + if not r: + break + b += r + bl += len(r) + # convert to log-encoding: + r = FileContainer.decode_line( + self.getFileName(), self.getEncoding(), b) + # ensure new-line is not in the middle (buffered 2 strings, e. g. in utf-16le it is "...[0A"+"00]..."): + e = r.find('\n') + if e >= 0 and e != len(r)-1: + l, r = r[0:e], r[0:e+1] + # back to bytes and get offset to seek after NL: + r = r.encode(self.getEncoding(), 'replace') + self.__handler.seek(-bl+len(r), 1) + return l + # trim new-line at end and check the line was written complete (contains a new-line): + l = r.rstrip('\r\n') + if l != r: + return l + if self.waitForLineEnd: + # not fulfilled - seek back and return: + self.__handler.seek(-bl, 1) + return None + return l def close(self): - if not self.__handler is None: - # Saves the last position. + if self.__handler is not None: + # Saves the last real position. self.__pos = self.__handler.tell() # Closes the file. self.__handler.close() self.__handler = None - ## print "D: Closed %s with pos %d" % (handler, self.__pos) - ## sys.stdout.flush() + + def __iter__(self): + return self + def next(self): + line = self.readline() + if line is None: + self.close() + raise StopIteration + return line _decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60); diff --git a/fail2ban/server/ipdns.py b/fail2ban/server/ipdns.py index d6dfbb9d..d917d031 100644 --- a/fail2ban/server/ipdns.py +++ b/fail2ban/server/ipdns.py @@ -257,6 +257,8 @@ class IPAddr(object): FAM_IPv6 = CIDR_RAW - socket.AF_INET6 def __new__(cls, ipstr, cidr=CIDR_UNSPEC): + if cidr == IPAddr.CIDR_UNSPEC and isinstance(ipstr, (tuple, list)): + cidr = IPAddr.CIDR_RAW if cidr == IPAddr.CIDR_RAW: # don't cache raw ip = super(IPAddr, cls).__new__(cls) ip.__init(ipstr, cidr) diff --git a/fail2ban/server/jail.py b/fail2ban/server/jail.py index 673b6454..2c84e475 100644 --- a/fail2ban/server/jail.py +++ b/fail2ban/server/jail.py @@ -295,7 +295,7 @@ class Jail(object): ): try: #logSys.debug('restored ticket: %s', ticket) - if self.filter.inIgnoreIPList(ticket.getIP(), log_ignore=True): continue + if self.filter.inIgnoreIPList(ticket.getID(), log_ignore=True): continue # mark ticked was restored from database - does not put it again into db: ticket.restored = True # correct start time / ban time (by the same end of ban): diff --git a/fail2ban/server/observer.py b/fail2ban/server/observer.py index b585706f..b1c9b37d 100644 --- a/fail2ban/server/observer.py +++ b/fail2ban/server/observer.py @@ -232,7 +232,7 @@ class ObserverThread(JailThread): if self._paused: continue else: - ## notify event deleted (shutdown) - just sleep a litle bit (waiting for shutdown events, prevent high cpu usage) + ## notify event deleted (shutdown) - just sleep a little bit (waiting for shutdown events, prevent high cpu usage) time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL) ## stop by shutdown and empty queue : if not self.is_full: @@ -364,7 +364,7 @@ class ObserverThread(JailThread): ## [Async] ban time increment functionality ... ## ----------------------------------------- - def failureFound(self, failManager, jail, ticket): + def failureFound(self, jail, ticket): """ Notify observer a failure for ip was found Observer will check ip was known (bad) and possibly increase an retry count @@ -372,7 +372,7 @@ class ObserverThread(JailThread): # check jail active : if not jail.isAlive() or not jail.getBanTimeExtra("increment"): return - ip = ticket.getIP() + ip = ticket.getID() unixTime = ticket.getTime() logSys.debug("[%s] Observer: failure found %s", jail.name, ip) # increase retry count for known (bad) ip, corresponding banCount of it (one try will count than 2, 3, 5, 9 ...) : @@ -380,7 +380,7 @@ class ObserverThread(JailThread): retryCount = 1 timeOfBan = None try: - maxRetry = failManager.getMaxRetry() + maxRetry = jail.filter.failManager.getMaxRetry() db = jail.database if db is not None: for banCount, timeOfBan, lastBanTime in db.getBan(ip, jail): @@ -403,18 +403,12 @@ class ObserverThread(JailThread): MyTime.time2str(unixTime), banCount, retryCount, (', Ban' if retryCount >= maxRetry else '')) # retryCount-1, because a ticket was already once incremented by filter self - retryCount = failManager.addFailure(ticket, retryCount - 1, True) + retryCount = jail.filter.failManager.addFailure(ticket, retryCount - 1, True) ticket.setBanCount(banCount) # after observe we have increased attempt count, compare it >= maxretry ... if retryCount >= maxRetry: # perform the banning of the IP now (again) - # [todo]: this code part will be used multiple times - optimize it later. - try: # pragma: no branch - exception is the only way out - while True: - ticket = failManager.toBan(ip) - jail.putFailTicket(ticket) - except FailManagerEmpty: - failManager.cleanup(MyTime.time()) + jail.filter.performBan(ip) except Exception as e: logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG) @@ -441,7 +435,7 @@ class ObserverThread(JailThread): if not jail.isAlive() or not jail.database: return banTime be = jail.getBanTimeExtra() - ip = ticket.getIP() + ip = ticket.getID() orgBanTime = banTime # check ip was already banned (increment time of ban): try: @@ -480,7 +474,7 @@ class ObserverThread(JailThread): return try: oldbtime = btime - ip = ticket.getIP() + ip = ticket.getID() logSys.debug("[%s] Observer: ban found %s, %s", jail.name, ip, btime) # if not permanent and ban time was not set - check time should be increased: if btime != -1 and ticket.getBanTime() is None: @@ -520,7 +514,7 @@ class ObserverThread(JailThread): """ try: btime = ticket.getBanTime() - ip = ticket.getIP() + ip = ticket.getID() logSys.debug("[%s] Observer: prolong %s, %s", jail.name, ip, btime) # prolong ticket via actions that expected this: jail.actions._prolongBan(ticket) diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index 3853bbc4..36ed1b0d 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -728,9 +728,7 @@ class Server: except (ValueError, KeyError): # pragma: no cover # Is known to be thrown after logging was shutdown once # with older Pythons -- seems to be safe to ignore there - # At least it was still failing on 2.6.2-0ubuntu1 (jaunty) - if (2, 6, 3) <= sys.version_info < (3,) or \ - (3, 2) <= sys.version_info: + if sys.version_info < (3,) or sys.version_info >= (3, 2): raise # detailed format by deep log levels (as DEBUG=10): if logger.getEffectiveLevel() <= logging.DEBUG: # pragma: no cover diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 6f88add1..12be163a 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -271,16 +271,12 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None): week_of_year = int(val) # U starts week on Sunday, W - on Monday week_of_year_start = 6 if key == 'U' else 0 - elif key == 'z': + elif key in ('z', 'Z'): z = val if z in ("Z", "UTC", "GMT"): tzoffset = 0 else: tzoffset = zone2offset(z, 0); # currently offset-based only - elif key == 'Z': - z = val - if z in ("UTC", "GMT"): - tzoffset = 0 # Fail2Ban will assume it's this year assume_year = False diff --git a/fail2ban/server/ticket.py b/fail2ban/server/ticket.py index f99b6462..96e67773 100644 --- a/fail2ban/server/ticket.py +++ b/fail2ban/server/ticket.py @@ -33,7 +33,7 @@ logSys = getLogger(__name__) class Ticket(object): - __slots__ = ('_ip', '_flags', '_banCount', '_banTime', '_time', '_data', '_retry', '_lastReset') + __slots__ = ('_id', '_flags', '_banCount', '_banTime', '_time', '_data', '_retry', '_lastReset') MAX_TIME = 0X7FFFFFFFFFFF ;# 4461763-th year @@ -48,7 +48,7 @@ class Ticket(object): @param matches (log) lines caused the ticket """ - self.setIP(ip) + self.setID(ip) self._flags = 0; self._banCount = 0; self._banTime = None; @@ -65,7 +65,7 @@ class Ticket(object): def __str__(self): return "%s: ip=%s time=%s bantime=%s bancount=%s #attempts=%d matches=%r" % \ - (self.__class__.__name__.split('.')[-1], self._ip, self._time, + (self.__class__.__name__.split('.')[-1], self._id, self._time, self._banTime, self._banCount, self._data['failures'], self._data.get('matches', [])) @@ -74,7 +74,7 @@ class Ticket(object): def __eq__(self, other): try: - return self._ip == other._ip and \ + return self._id == other._id and \ round(self._time, 2) == round(other._time, 2) and \ self._data == other._data except AttributeError: @@ -86,18 +86,17 @@ class Ticket(object): if v is not None: setattr(self, n, v) - - def setIP(self, value): + def setID(self, value): # guarantee using IPAddr instead of unicode, str for the IP if isinstance(value, basestring): value = IPAddr(value) - self._ip = value + self._id = value def getID(self): - return self._data.get('fid', self._ip) + return self._id def getIP(self): - return self._ip + return self._data.get('ip', self._id) def setTime(self, value): self._time = value diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py index 294d147f..18073ea7 100644 --- a/fail2ban/server/utils.py +++ b/fail2ban/server/utils.py @@ -30,11 +30,7 @@ import sys from threading import Lock import time from ..helpers import getLogger, _merge_dicts, uni_decode - -try: - from collections import OrderedDict -except ImportError: # pragma: 3.x no cover - OrderedDict = dict +from collections import OrderedDict if sys.version_info >= (3, 3): import importlib.machinery @@ -100,24 +96,12 @@ class Utils(): with self.__lock: # clean cache if max count reached: if len(cache) >= self.maxCount: - if OrderedDict is not dict: - # ordered (so remove some from ahead, FIFO) - while cache: - (ck, cv) = cache.popitem(last=False) - # if not yet expired (but has free slot for new entry): - if cv[1] > t and len(cache) < self.maxCount: - break - else: # pragma: 3.x no cover (dict is in 2.6 only) - remlst = [] - for (ck, cv) in cache.iteritems(): - # if expired: - if cv[1] <= t: - remlst.append(ck) - for ck in remlst: - self._cache.pop(ck, None) - # if still max count - remove any one: - while cache and len(cache) >= self.maxCount: - cache.popitem() + # ordered (so remove some from ahead, FIFO) + while cache: + (ck, cv) = cache.popitem(last=False) + # if not yet expired (but has free slot for new entry): + if cv[1] > t and len(cache) < self.maxCount: + break # set now: cache[k] = (v, t + self.maxTime) @@ -332,11 +316,9 @@ class Utils(): timeout_expr = lambda: time.time() > time0 else: timeout_expr = timeout - if not interval: - interval = Utils.DEFAULT_SLEEP_INTERVAL if timeout_expr(): break - stm = min(stm + interval, Utils.DEFAULT_SLEEP_TIME) + stm = min(stm + (interval or Utils.DEFAULT_SLEEP_INTERVAL), Utils.DEFAULT_SLEEP_TIME) time.sleep(stm) return ret diff --git a/fail2ban/tests/actionstestcase.py b/fail2ban/tests/actionstestcase.py index 7b85ff94..9c9add65 100644 --- a/fail2ban/tests/actionstestcase.py +++ b/fail2ban/tests/actionstestcase.py @@ -217,6 +217,9 @@ class ExecuteActions(LogCaptureTestCase): # flush for inet6 is intentionally "broken" here - test no unhandled except and invariant check: act['actionflush?family=inet6'] = act.actionflush + '; exit 1' act.actionstart_on_demand = True + # force errors via check in ban/unban: + act.actionban = "<actioncheck> ; " + act.actionban + act.actionunban = "<actioncheck> ; " + act.actionunban self.__actions.start() self.assertNotLogged("stdout: %r" % 'ip start') @@ -294,6 +297,9 @@ class ExecuteActions(LogCaptureTestCase): act['actionflush?family=inet6'] = act.actionflush + '; exit 1' act.actionstart_on_demand = True act.actionrepair_on_unban = True + # force errors via check in ban/unban: + act.actionban = "<actioncheck> ; " + act.actionban + act.actionunban = "<actioncheck> ; " + act.actionunban self.__actions.start() self.assertNotLogged("stdout: %r" % 'ip start') diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py index d45c3171..ce5de483 100644 --- a/fail2ban/tests/actiontestcase.py +++ b/fail2ban/tests/actiontestcase.py @@ -75,61 +75,59 @@ class CommandActionTest(LogCaptureTestCase): lambda: substituteRecursiveTags({'A': 'to=<B> fromip=<IP>', 'C': '<B>', 'B': '<C>', 'D': ''})) self.assertRaises(ValueError, lambda: substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP>', 'sweet': '<honeypot>', 'honeypot': '<sweet>', 'ignoreregex': ''})) - # We need here an ordered, because the sequence of iteration is very important for this test - if OrderedDict: - # No cyclic recursion, just multiple replacement of tag <T>, should be successful: - self.assertEqual(substituteRecursiveTags( OrderedDict( - (('X', 'x=x<T>'), ('T', '1'), ('Z', '<X> <T> <Y>'), ('Y', 'y=y<T>'))) - ), {'X': 'x=x1', 'T': '1', 'Y': 'y=y1', 'Z': 'x=x1 1 y=y1'} - ) - # No cyclic recursion, just multiple replacement of tag <T> in composite tags, should be successful: - self.assertEqual(substituteRecursiveTags( OrderedDict( - (('X', 'x=x<T> <Z> <<R1>> <<R2>>'), ('R1', 'Z'), ('R2', 'Y'), ('T', '1'), ('Z', '<T> <Y>'), ('Y', 'y=y<T>'))) - ), {'X': 'x=x1 1 y=y1 1 y=y1 y=y1', 'R1': 'Z', 'R2': 'Y', 'T': '1', 'Z': '1 y=y1', 'Y': 'y=y1'} - ) - # No cyclic recursion, just multiple replacement of same tags, should be successful: - self.assertEqual(substituteRecursiveTags( OrderedDict(( - ('actionstart', 'ipset create <ipmset> hash:ip timeout <bantime> family <ipsetfamily>\n<iptables> -I <chain> <actiontype>'), - ('ipmset', 'f2b-<name>'), - ('name', 'any'), - ('bantime', '600'), - ('ipsetfamily', 'inet'), - ('iptables', 'iptables <lockingopt>'), - ('lockingopt', '-w'), - ('chain', 'INPUT'), - ('actiontype', '<multiport>'), - ('multiport', '-p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>'), - ('protocol', 'tcp'), - ('port', 'ssh'), - ('blocktype', 'REJECT',), - )) - ), OrderedDict(( - ('actionstart', 'ipset create f2b-any hash:ip timeout 600 family inet\niptables -w -I INPUT -p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), - ('ipmset', 'f2b-any'), - ('name', 'any'), - ('bantime', '600'), - ('ipsetfamily', 'inet'), - ('iptables', 'iptables -w'), - ('lockingopt', '-w'), - ('chain', 'INPUT'), - ('actiontype', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), - ('multiport', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), - ('protocol', 'tcp'), - ('port', 'ssh'), - ('blocktype', 'REJECT') - )) - ) - # Cyclic recursion by composite tag creation, tags "create" another tag, that closes cycle: - self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( - ('A', '<<B><C>>'), - ('B', 'D'), ('C', 'E'), - ('DE', 'cycle <A>'), - )) )) - self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( - ('DE', 'cycle <A>'), - ('A', '<<B><C>>'), - ('B', 'D'), ('C', 'E'), - )) )) + # No cyclic recursion, just multiple replacement of tag <T>, should be successful: + self.assertEqual(substituteRecursiveTags( OrderedDict( + (('X', 'x=x<T>'), ('T', '1'), ('Z', '<X> <T> <Y>'), ('Y', 'y=y<T>'))) + ), {'X': 'x=x1', 'T': '1', 'Y': 'y=y1', 'Z': 'x=x1 1 y=y1'} + ) + # No cyclic recursion, just multiple replacement of tag <T> in composite tags, should be successful: + self.assertEqual(substituteRecursiveTags( OrderedDict( + (('X', 'x=x<T> <Z> <<R1>> <<R2>>'), ('R1', 'Z'), ('R2', 'Y'), ('T', '1'), ('Z', '<T> <Y>'), ('Y', 'y=y<T>'))) + ), {'X': 'x=x1 1 y=y1 1 y=y1 y=y1', 'R1': 'Z', 'R2': 'Y', 'T': '1', 'Z': '1 y=y1', 'Y': 'y=y1'} + ) + # No cyclic recursion, just multiple replacement of same tags, should be successful: + self.assertEqual(substituteRecursiveTags( OrderedDict(( + ('actionstart', 'ipset create <ipmset> hash:ip timeout <bantime> family <ipsetfamily>\n<iptables> -I <chain> <actiontype>'), + ('ipmset', 'f2b-<name>'), + ('name', 'any'), + ('bantime', '600'), + ('ipsetfamily', 'inet'), + ('iptables', 'iptables <lockingopt>'), + ('lockingopt', '-w'), + ('chain', 'INPUT'), + ('actiontype', '<multiport>'), + ('multiport', '-p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>'), + ('protocol', 'tcp'), + ('port', 'ssh'), + ('blocktype', 'REJECT',), + )) + ), OrderedDict(( + ('actionstart', 'ipset create f2b-any hash:ip timeout 600 family inet\niptables -w -I INPUT -p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), + ('ipmset', 'f2b-any'), + ('name', 'any'), + ('bantime', '600'), + ('ipsetfamily', 'inet'), + ('iptables', 'iptables -w'), + ('lockingopt', '-w'), + ('chain', 'INPUT'), + ('actiontype', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), + ('multiport', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'), + ('protocol', 'tcp'), + ('port', 'ssh'), + ('blocktype', 'REJECT') + )) + ) + # Cyclic recursion by composite tag creation, tags "create" another tag, that closes cycle: + self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( + ('A', '<<B><C>>'), + ('B', 'D'), ('C', 'E'), + ('DE', 'cycle <A>'), + )) )) + self.assertRaises(ValueError, lambda: substituteRecursiveTags( OrderedDict(( + ('DE', 'cycle <A>'), + ('A', '<<B><C>>'), + ('B', 'D'), ('C', 'E'), + )) )) # missing tags are ok self.assertEqual(substituteRecursiveTags({'A': '<C>'}), {'A': '<C>'}) @@ -305,8 +303,8 @@ class CommandActionTest(LogCaptureTestCase): self.assertEqual(self.__action.actionstart, "touch '%s'" % tmp) self.__action.actionstop = "rm -f '%s'" % tmp self.assertEqual(self.__action.actionstop, "rm -f '%s'" % tmp) - self.__action.actionban = "echo -n" - self.assertEqual(self.__action.actionban, 'echo -n') + self.__action.actionban = "<actioncheck> && echo -n" + self.assertEqual(self.__action.actionban, "<actioncheck> && echo -n") self.__action.actioncheck = "[ -e '%s' ]" % tmp self.assertEqual(self.__action.actioncheck, "[ -e '%s' ]" % tmp) self.__action.actionunban = "true" @@ -316,6 +314,7 @@ class CommandActionTest(LogCaptureTestCase): self.assertNotLogged('returned') # no action was actually executed yet + # start on demand is false, so it should cause failure on first attempt of ban: self.__action.ban({'ip': None}) self.assertLogged('Invariant check failed') self.assertLogged('returned successfully') @@ -365,13 +364,52 @@ class CommandActionTest(LogCaptureTestCase): self.pruneLog('[phase 2]') self.__action.actionstart = "touch '%s'" % tmp self.__action.actionstop = "rm '%s'" % tmp - self.__action.actionban = """printf "%%%%b\n" <ip> >> '%s'""" % tmp + self.__action.actionban = """<actioncheck> && printf "%%%%b\n" <ip> >> '%s'""" % tmp self.__action.actioncheck = "[ -e '%s' ]" % tmp self.__action.ban({'ip': None}) self.assertLogged('Invariant check failed') self.assertNotLogged('Unable to restore environment') @with_tmpdir + def testExecuteActionCheckOnBanFailure(self, tmp): + tmp += '/fail2ban.test' + self.__action.actionstart = "touch '%s'; echo 'started ...'" % tmp + self.__action.actionstop = "rm -f '%s'" % tmp + self.__action.actionban = "[ -e '%s' ] && echo 'banned '<ip>" % tmp + self.__action.actioncheck = "[ -e '%s' ] && echo 'check ok' || { echo 'check failed'; exit 1; }" % tmp + self.__action.actionrepair = "echo 'repair ...'; touch '%s'" % tmp + self.__action.actionstart_on_demand = False + self.__action.start() + # phase 1: with repair; + # phase 2: without repair (start/stop), not on demand; + # phase 3: without repair (start/stop), start on demand. + for i in (1, 2, 3): + self.pruneLog('[phase %s]' % i) + # 1st time with success ban: + self.__action.ban({'ip': '192.0.2.1'}) + self.assertLogged( + "stdout: %r" % 'banned 192.0.2.1', all=True) + self.assertNotLogged("Invariant check failed. Trying", + "stdout: %r" % 'check failed', + "stdout: %r" % ('repair ...' if self.__action.actionrepair else 'started ...'), + "stdout: %r" % 'check ok', all=True) + # force error in ban: + os.remove(tmp) + self.pruneLog() + # 2nd time with fail recognition, success repair, check and ban: + self.__action.ban({'ip': '192.0.2.2'}) + self.assertLogged("Invariant check failed. Trying", + "stdout: %r" % 'check failed', + "stdout: %r" % ('repair ...' if self.__action.actionrepair else 'started ...'), + "stdout: %r" % 'check ok', + "stdout: %r" % 'banned 192.0.2.2', all=True) + # repeat without repair (stop/start), herafter enable on demand: + if self.__action.actionrepair: + self.__action.actionrepair = "" + elif not self.__action.actionstart_on_demand: + self.__action.actionstart_on_demand = True + + @with_tmpdir def testExecuteActionCheckRepairEnvironment(self, tmp): tmp += '/fail2ban.test' self.__action.actionstart = "" diff --git a/fail2ban/tests/banmanagertestcase.py b/fail2ban/tests/banmanagertestcase.py index ec8e6f9f..cf25ac0f 100644 --- a/fail2ban/tests/banmanagertestcase.py +++ b/fail2ban/tests/banmanagertestcase.py @@ -100,23 +100,23 @@ class AddFailure(unittest.TestCase): self.assertFalse(self.__banManager._inBanList(ticket)) def testBanTimeIncr(self): - ticket = BanTicket(self.__ticket.getIP(), self.__ticket.getTime()) + ticket = BanTicket(self.__ticket.getID(), self.__ticket.getTime()) ## increase twice and at end permanent, check time/count increase: c = 0 for i in (1000, 2000, -1): self.__banManager.addBanTicket(self.__ticket); c += 1 ticket.setBanTime(i) self.assertFalse(self.__banManager.addBanTicket(ticket)); # no incr of c (already banned) - self.assertEqual(str(self.__banManager.getTicketByID(ticket.getIP())), - "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getIP(), ticket.getTime(), i, c)) + self.assertEqual(str(self.__banManager.getTicketByID(ticket.getID())), + "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getID(), ticket.getTime(), i, c)) ## after permanent, it should remain permanent ban time (-1): self.__banManager.addBanTicket(self.__ticket); c += 1 ticket.setBanTime(-1) self.assertFalse(self.__banManager.addBanTicket(ticket)); # no incr of c (already banned) ticket.setBanTime(1000) self.assertFalse(self.__banManager.addBanTicket(ticket)); # no incr of c (already banned) - self.assertEqual(str(self.__banManager.getTicketByID(ticket.getIP())), - "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getIP(), ticket.getTime(), -1, c)) + self.assertEqual(str(self.__banManager.getTicketByID(ticket.getID())), + "BanTicket: ip=%s time=%s bantime=%s bancount=%s #attempts=0 matches=[]" % (ticket.getID(), ticket.getTime(), -1, c)) def testUnban(self): btime = self.__banManager.getBanTime() diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index 4029c753..37083a06 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -698,7 +698,7 @@ class JailsReaderTestCache(LogCaptureTestCase): cnt = self._getLoggedReadCount(r'filter\.d/common\.conf') self.assertTrue(cnt == 1, "Unexpected count by reading of filter files, cnt = %s" % cnt) # same with action: - cnt = self._getLoggedReadCount(r'action\.d/iptables-common\.conf') + cnt = self._getLoggedReadCount(r'action\.d/iptables\.conf') self.assertTrue(cnt == 1, "Unexpected count by reading of action files, cnt = %s" % cnt) finally: configparserinc.logLevel = saved_ll diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py index a8e2ceae..8cc394be 100644 --- a/fail2ban/tests/databasetestcase.py +++ b/fail2ban/tests/databasetestcase.py @@ -29,7 +29,7 @@ import tempfile import sqlite3 import shutil -from ..server.filter import FileContainer +from ..server.filter import FileContainer, Filter from ..server.mytime import MyTime from ..server.ticket import FailTicket from ..server.actions import Actions, Utils @@ -192,7 +192,7 @@ class DatabaseTest(LogCaptureTestCase): ticket.setAttempt(3) self.assertEqual(bans[0], ticket) # second ban found also: - self.assertEqual(bans[1].getIP(), "1.2.3.8") + self.assertEqual(bans[1].getID(), "1.2.3.8") # updated ? self.assertEqual(self.db.updateDb(Fail2BanDb.__version__), Fail2BanDb.__version__) # check current bans (should find 2 tickets after upgrade): @@ -212,19 +212,20 @@ class DatabaseTest(LogCaptureTestCase): self.jail.name in self.db.getJailNames(True), "Jail not added to database") - def testAddLog(self): + def _testAddLog(self): self.testAddJail() # Jail required _, filename = tempfile.mkstemp(".log", "Fail2BanDb_") self.fileContainer = FileContainer(filename, "utf-8") - self.db.addLog(self.jail, self.fileContainer) + pos = self.db.addLog(self.jail, self.fileContainer) + self.assertTrue(pos is None); # unknown previously self.assertIn(filename, self.db.getLogPaths(self.jail)) os.remove(filename) def testUpdateLog(self): - self.testAddLog() # Add log file + self._testAddLog() # Add log file # Write some text filename = self.fileContainer.getFileName() @@ -311,7 +312,7 @@ class DatabaseTest(LogCaptureTestCase): for i, ticket in enumerate(tickets): DefLogSys.debug('readtickets[%d]: %r', i, readtickets[i].getData()) DefLogSys.debug(' == tickets[%d]: %r', i, ticket.getData()) - self.assertEqual(readtickets[i].getIP(), ticket.getIP()) + self.assertEqual(readtickets[i].getID(), ticket.getID()) self.assertEqual(len(readtickets[i].getMatches()), len(ticket.getMatches())) self.pruneLog('[test-phase 2] simulate errors') @@ -353,10 +354,10 @@ class DatabaseTest(LogCaptureTestCase): def testDelBan(self): tickets = self._testAdd3Bans() # delete single IP: - self.db.delBan(self.jail, tickets[0].getIP()) + self.db.delBan(self.jail, tickets[0].getID()) self.assertEqual(len(self.db.getBans(jail=self.jail)), 2) # delete two IPs: - self.db.delBan(self.jail, tickets[1].getIP(), tickets[2].getIP()) + self.db.delBan(self.jail, tickets[1].getID(), tickets[2].getID()) self.assertEqual(len(self.db.getBans(jail=self.jail)), 0) def testFlushBans(self): @@ -397,7 +398,7 @@ class DatabaseTest(LogCaptureTestCase): # should retrieve 2 matches only, but count of all attempts: self.db.maxMatches = maxMatches; ticket = self.db.getBansMerged("127.0.0.1") - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") self.assertEqual(ticket.getAttempt(), len(failures)) self.assertEqual(len(ticket.getMatches()), maxMatches) self.assertEqual(ticket.getMatches(), matches2find[-maxMatches:]) @@ -455,13 +456,13 @@ class DatabaseTest(LogCaptureTestCase): # All for IP 127.0.0.1 ticket = self.db.getBansMerged("127.0.0.1") - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") self.assertEqual(ticket.getAttempt(), 70) self.assertEqual(ticket.getMatches(), ["abc\n", "123\n", "ABC\n"]) # All for IP 127.0.0.1 for single jail ticket = self.db.getBansMerged("127.0.0.1", jail=self.jail) - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") self.assertEqual(ticket.getAttempt(), 30) self.assertEqual(ticket.getMatches(), ["abc\n", "123\n"]) @@ -489,8 +490,8 @@ class DatabaseTest(LogCaptureTestCase): tickets = self.db.getBansMerged() self.assertEqual(len(tickets), 2) self.assertSortedEqual( - list(set(ticket.getIP() for ticket in tickets)), - [ticket.getIP() for ticket in tickets]) + list(set(ticket.getID() for ticket in tickets)), + [ticket.getID() for ticket in tickets]) tickets = self.db.getBansMerged(jail=jail2) self.assertEqual(len(tickets), 1) @@ -509,7 +510,7 @@ class DatabaseTest(LogCaptureTestCase): tickets = self.db.getCurrentBans(jail=self.jail) self.assertEqual(len(tickets), 2) ticket = self.db.getCurrentBans(jail=None, ip="127.0.0.1"); - self.assertEqual(ticket.getIP(), "127.0.0.1") + self.assertEqual(ticket.getID(), "127.0.0.1") # positive case (1 ticket not yet expired): tickets = self.db.getCurrentBans(jail=self.jail, forbantime=15, @@ -544,17 +545,21 @@ class DatabaseTest(LogCaptureTestCase): self.testAddJail() # Jail required self.jail.database = self.db self.db.addJail(self.jail) - actions = Actions(self.jail) + actions = self.jail.actions actions.add( "action_checkainfo", os.path.join(TEST_FILES_DIR, "action.d/action_checkainfo.py"), {}) + actions.banManager.setBanTotal(20) + self.jail._Jail__filter = flt = Filter(self.jail) + flt.failManager.setFailTotal(50) ticket = FailTicket("1.2.3.4") ticket.setAttempt(5) ticket.setMatches(['test', 'test']) self.jail.putFailTicket(ticket) actions._Actions__checkBan() self.assertLogged("ban ainfo %s, %s, %s, %s" % (True, True, True, True)) + self.assertLogged("jail info %d, %d, %d, %d" % (1, 21, 0, 50)) def testDelAndAddJail(self): self.testAddJail() # Add jail diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index b8e8451e..83dd2671 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -516,6 +516,9 @@ class CustomDateFormatsTest(unittest.TestCase): (1072746123.0 - 3600, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03] server ..."), (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 UTC] server ..."), (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 UTC] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 Z] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 +0000] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 Z] server ..."), ): logSys.debug('== test: %r', (matched, dp, line)) if dp is None: diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index 86480f02..66cbf6da 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -230,7 +230,7 @@ def _start_params(tmp, use_stock=False, use_stock_cfg=None, os.symlink(os.path.abspath(pjoin(STOCK_CONF_DIR, n)), pjoin(cfg, n)) if create_before_start: for n in create_before_start: - _write_file(n % {'tmp': tmp}, 'w', '') + _write_file(n % {'tmp': tmp}, 'w') # parameters (sock/pid and config, increase verbosity, set log, etc.): vvv, llev = (), "INFO" if unittest.F2B.log_level < logging.INFO: # pragma: no cover @@ -942,10 +942,8 @@ class Fail2banServerTest(Fail2banClientServerBase): "Jail 'broken-jail' skipped, because of wrong configuration", all=True) # enable both jails, 3 logs for jail1, etc... - # truncate test-log - we should not find unban/ban again by reload: self.pruneLog("[test-phase 1b]") _write_jail_cfg(actions=[1,2]) - _write_file(test1log, "w+") if unittest.F2B.log_level < logging.DEBUG: # pragma: no cover _out_file(test1log) self.execCmd(SUCCESS, startparams, "reload") @@ -1008,7 +1006,7 @@ class Fail2banServerTest(Fail2banClientServerBase): self.pruneLog("[test-phase 2b]") # write new failures: - _write_file(test2log, "w+", *( + _write_file(test2log, "a+", *( (str(int(MyTime.time())) + " error 403 from 192.0.2.2: test 2",) * 3 + (str(int(MyTime.time())) + " error 403 from 192.0.2.3: test 2",) * 3 + (str(int(MyTime.time())) + " failure 401 from 192.0.2.4: test 2",) * 3 + @@ -1067,10 +1065,6 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertEqual(self.execCmdDirect(startparams, 'get', 'test-jail1', 'banned', '192.0.2.3', '192.0.2.9')[1], [1, 0]) - # rotate logs: - _write_file(test1log, "w+") - _write_file(test2log, "w+") - # restart jail without unban all: self.pruneLog("[test-phase 2c]") self.execCmd(SUCCESS, startparams, @@ -1188,7 +1182,7 @@ class Fail2banServerTest(Fail2banClientServerBase): # now write failures again and check already banned (jail1 was alive the whole time) and new bans occurred (jail1 was alive the whole time): self.pruneLog("[test-phase 5]") - _write_file(test1log, "w+", *( + _write_file(test1log, "a+", *( (str(int(MyTime.time())) + " failure 401 from 192.0.2.1: test 5",) * 3 + (str(int(MyTime.time())) + " error 403 from 192.0.2.5: test 5",) * 3 + (str(int(MyTime.time())) + " failure 401 from 192.0.2.6: test 5",) * 3 @@ -1415,8 +1409,9 @@ class Fail2banServerTest(Fail2banClientServerBase): 'jails': ( # default: '''test_action = dummy[actionstart_on_demand=1, init="start: %(__name__)s", target="%(tmp)s/test.txt", - actionban='<known/actionban>; - echo "<matches>"; printf "=====\\n%%b\\n=====\\n\\n" "<matches>" >> <target>']''', + actionban='<known/actionban>; echo "found: <jail.found> / <jail.found_total>, banned: <jail.banned> / <jail.banned_total>" + echo "<matches>"; printf "=====\\n%%b\\n=====\\n\\n" "<matches>" >> <target>', + actionstop='<known/actionstop>; echo "stats <name> - found: <jail.found_total>, banned: <jail.banned_total>"']''', # jail sendmail-auth: '[sendmail-auth]', 'backend = polling', @@ -1461,7 +1456,8 @@ class Fail2banServerTest(Fail2banClientServerBase): _write_file(lgfn, "w+", *smaut_msg) # wait and check it caused banned (and dump in the test-file): self.assertLogged( - "[sendmail-auth] Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME) + "[sendmail-auth] Ban 192.0.2.1", "stdout: 'found: 0 / 3, banned: 1 / 1'", + "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME) _out_file(tofn) td = _read_file(tofn) # check matches (maxmatches = 2, so only 2 & 3 available): @@ -1472,10 +1468,11 @@ class Fail2banServerTest(Fail2banClientServerBase): self.pruneLog("[test-phase sendmail-reject]") # write log: - _write_file(lgfn, "w+", *smrej_msg) + _write_file(lgfn, "a+", *smrej_msg) # wait and check it caused banned (and dump in the test-file): self.assertLogged( - "[sendmail-reject] Ban 192.0.2.2", "1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME) + "[sendmail-reject] Ban 192.0.2.2", "stdout: 'found: 0 / 3, banned: 1 / 1'", + "1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME) _out_file(tofn) td = _read_file(tofn) # check matches (no maxmatches, so all matched messages are available): @@ -1489,6 +1486,8 @@ class Fail2banServerTest(Fail2banClientServerBase): # wait a bit: self.assertLogged( "Reload finished.", + "stdout: 'stats sendmail-auth - found: 3, banned: 1'", + "stdout: 'stats sendmail-reject - found: 3, banned: 1'", "[sendmail-auth] Restore Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME) # check matches again - (dbmaxmatches = 1), so it should be only last match after restart: td = _read_file(tofn) @@ -1597,7 +1596,7 @@ class Fail2banServerTest(Fail2banClientServerBase): wakeObs = False _observer_wait_before_incrban(lambda: wakeObs) # write again (IP already bad): - _write_file(test1log, "w+", *( + _write_file(test1log, "a+", *( (str(int(MyTime.time())) + " failure 401 from 192.0.2.11: I'm very bad \"hacker\" `` $(echo test)",) * 2 )) # wait for ban: diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 884f313a..e300f315 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -25,6 +25,7 @@ __license__ = "GPL" import os import sys +import tempfile import unittest from ..client import fail2banregex @@ -80,6 +81,11 @@ def _test_exec_command_line(*args): sys.stderr = _org['stderr'] return _exit_code +def _reset(): + # reset global warn-counter: + from ..server.filter import _decode_line_warn + _decode_line_warn.clear() + STR_00 = "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0" STR_00_NODT = "[sshd] error: PAM: Authentication failure for kevin from 192.0.2.0" @@ -122,6 +128,7 @@ class Fail2banRegexTest(LogCaptureTestCase): """Call before every test case.""" LogCaptureTestCase.setUp(self) setUpMyTime() + _reset() def tearDown(self): """Call after every test case.""" @@ -348,22 +355,35 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('kevin') self.pruneLog() # multiple id combined to a tuple (id, tuple_id): - self.assertTrue(_test_exec('-o', 'id', + self.assertTrue(_test_exec('-o', 'id', '-d', '{^LN-BEG}EPOCH', '1591983743.667 192.0.2.1 192.0.2.2', r'^\s*<F-ID/> <F-TUPLE_ID>\S+</F-TUPLE_ID>')) self.assertLogged(str(('192.0.2.1', '192.0.2.2'))) self.pruneLog() # multiple id combined to a tuple, id first - (id, tuple_id_1, tuple_id_2): - self.assertTrue(_test_exec('-o', 'id', + self.assertTrue(_test_exec('-o', 'id', '-d', '{^LN-BEG}EPOCH', '1591983743.667 left 192.0.2.3 right', r'^\s*<F-TUPLE_ID_1>\S+</F-TUPLE_ID_1> <F-ID/> <F-TUPLE_ID_2>\S+</F-TUPLE_ID_2>')) + self.assertLogged(str(('192.0.2.3', 'left', 'right'))) self.pruneLog() # id had higher precedence as ip-address: - self.assertTrue(_test_exec('-o', 'id', + self.assertTrue(_test_exec('-o', 'id', '-d', '{^LN-BEG}EPOCH', '1591983743.667 left [192.0.2.4]:12345 right', r'^\s*<F-TUPLE_ID_1>\S+</F-TUPLE_ID_1> <F-ID><ADDR>:<F-PORT/></F-ID> <F-TUPLE_ID_2>\S+</F-TUPLE_ID_2>')) self.assertLogged(str(('[192.0.2.4]:12345', 'left', 'right'))) self.pruneLog() + # ip is not id anymore (if IP-address deviates from ID): + self.assertTrue(_test_exec('-o', 'ip', '-d', '{^LN-BEG}EPOCH', + '1591983743.667 left [192.0.2.4]:12345 right', + r'^\s*<F-TUPLE_ID_1>\S+</F-TUPLE_ID_1> <F-ID><ADDR>:<F-PORT/></F-ID> <F-TUPLE_ID_2>\S+</F-TUPLE_ID_2>')) + self.assertNotLogged(str(('[192.0.2.4]:12345', 'left', 'right'))) + self.assertLogged('192.0.2.4') + self.pruneLog() + self.assertTrue(_test_exec('-o', 'ID:<fid> | IP:<ip>', '-d', '{^LN-BEG}EPOCH', + '1591983743.667 left [192.0.2.4]:12345 right', + r'^\s*<F-TUPLE_ID_1>\S+</F-TUPLE_ID_1> <F-ID><ADDR>:<F-PORT/></F-ID> <F-TUPLE_ID_2>\S+</F-TUPLE_ID_2>')) + self.assertLogged('ID:'+str(('[192.0.2.4]:12345', 'left', 'right'))+' | IP:192.0.2.4') + self.pruneLog() # row with id : self.assertTrue(_test_exec('-o', 'row', STR_00, RE_00_ID)) self.assertLogged("['kevin'", "'ip4': '192.0.2.0'", "'fid': 'kevin'", all=True) @@ -385,6 +405,43 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('192.0.2.0, kevin, inet4') self.pruneLog() + def testStalledIPByNoFailFrmtOutput(self): + opts = ( + '-c', CONFIG_DIR, + "-d", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", + ) + log = ( + 'May 27 00:16:33 host sshd[2364]: User root not allowed because account is locked\n' + 'May 27 00:16:33 host sshd[2364]: Received disconnect from 192.0.2.76 port 58846:11: Bye Bye [preauth]' + ) + _test = lambda *args: _test_exec(*(opts + args)) + # with MLFID from prefregex and IP after failure obtained from F-NOFAIL RE: + self.assertTrue(_test('-o', 'IP:<ip>', log, 'sshd')) + self.assertLogged('IP:192.0.2.76') + self.pruneLog() + # test diverse ID/IP constellations: + def _test_variants(flt="sshd", prefix=""): + # with different ID/IP from failregex (ID/User from first, IP from second message): + self.assertTrue(_test('-o', 'ID:"<fid>" | IP:<ip> | U:<F-USER>', log, + flt+'[failregex="' + '^'+prefix+'<F-ID>User <F-USER>\S+</F-USER></F-ID> not allowed\n' + '^'+prefix+'Received disconnect from <ADDR>' + '"]')) + self.assertLogged('ID:"User root" | IP:192.0.2.76 | U:root') + self.pruneLog() + # with different ID/IP from failregex (User from first, ID and IP from second message): + self.assertTrue(_test('-o', 'ID:"<fid>" | IP:<ip> | U:<F-USER>', log, + flt+'[failregex="' + '^'+prefix+'User <F-USER>\S+</F-USER> not allowed\n' + '^'+prefix+'Received disconnect from <F-ID><ADDR> port \d+</F-ID>' + '"]')) + self.assertLogged('ID:"192.0.2.76 port 58846" | IP:192.0.2.76 | U:root') + self.pruneLog() + # first with sshd and prefregex: + _test_variants() + # the same without prefregex and MLFID directly in failregex (no merge with prefregex groups): + _test_variants('common', prefix="\s*\S+ sshd\[<F-MLFID>\d+</F-MLFID>\]:\s+") + def testNoDateTime(self): # datepattern doesn't match: self.assertTrue(_test_exec('-d', '{^LN-BEG}EPOCH', '-o', 'Found-ID:<F-ID>', STR_00_NODT, RE_00_ID)) @@ -485,14 +542,8 @@ class Fail2banRegexTest(LogCaptureTestCase): FILENAME_ZZZ_GEN, FILENAME_ZZZ_GEN )) - def _reset(self): - # reset global warn-counter: - from ..server.filter import _decode_line_warn - _decode_line_warn.clear() - def testWronChar(self): unittest.F2B.SkipIfCfgMissing(stock=True) - self._reset() self.assertTrue(_test_exec( "-l", "notice", # put down log-level, because of too many debug-messages "--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", @@ -508,7 +559,6 @@ class Fail2banRegexTest(LogCaptureTestCase): def testWronCharDebuggex(self): unittest.F2B.SkipIfCfgMissing(stock=True) - self._reset() self.assertTrue(_test_exec( "-l", "notice", # put down log-level, because of too many debug-messages "--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", @@ -521,6 +571,36 @@ class Fail2banRegexTest(LogCaptureTestCase): self.assertLogged('https://') + def testNLCharAsPartOfUniChar(self): + fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni') + # test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02): + for enc in ('utf-16be', 'utf-16le'): + self.pruneLog("[test-phase encoding=%s]" % enc) + try: + fout = open(fname, 'wb') + # test on unicode string containing \x0A as part of uni-char, + # it must produce exactly 2 lines (both are failures): + for l in ( + u'1490349000 \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n', + u'1490349000 \u20AC Failed auth: invalid user TestI from 192.0.2.2\n' + ): + fout.write(l.encode(enc)) + fout.close() + + self.assertTrue(_test_exec( + "-l", "notice", # put down log-level, because of too many debug-messages + "--encoding", enc, + "--datepattern", r"^EPOCH", + fname, r"Failed .* from <HOST>", + )) + + self.assertLogged(" encoding : %s" % enc, + "Lines: 2 lines, 0 ignored, 2 matched, 0 missed", all=True) + self.assertNotLogged("Missed line(s)") + finally: + fout.close() + os.unlink(fname) + def testExecCmdLine_Usage(self): self.assertNotEqual(_test_exec_command_line(), 0) self.pruneLog() diff --git a/fail2ban/tests/failmanagertestcase.py b/fail2ban/tests/failmanagertestcase.py index a5425286..42b0fbd2 100644 --- a/fail2ban/tests/failmanagertestcase.py +++ b/fail2ban/tests/failmanagertestcase.py @@ -150,8 +150,8 @@ class AddFailure(unittest.TestCase): self.__failManager.setMaxRetry(5) #ticket = FailTicket('193.168.0.128', None) ticket = self.__failManager.toBan() - self.assertEqual(ticket.getIP(), "193.168.0.128") - self.assertTrue(isinstance(ticket.getIP(), (str, IPAddr))) + self.assertEqual(ticket.getID(), "193.168.0.128") + self.assertTrue(isinstance(ticket.getID(), (str, IPAddr))) # finish with rudimentary tests of the ticket # verify consistent str @@ -180,9 +180,9 @@ class AddFailure(unittest.TestCase): def testWindow(self): self._addDefItems() ticket = self.__failManager.toBan() - self.assertNotEqual(ticket.getIP(), "100.100.10.10") + self.assertNotEqual(ticket.getID(), "100.100.10.10") ticket = self.__failManager.toBan() - self.assertNotEqual(ticket.getIP(), "100.100.10.10") + self.assertNotEqual(ticket.getID(), "100.100.10.10") self.assertRaises(FailManagerEmpty, self.__failManager.toBan) def testBgService(self): diff --git a/fail2ban/tests/files/action.d/action_checkainfo.py b/fail2ban/tests/files/action.d/action_checkainfo.py index 63dd4f5b..c5eaf0f8 100644 --- a/fail2ban/tests/files/action.d/action_checkainfo.py +++ b/fail2ban/tests/files/action.d/action_checkainfo.py @@ -8,6 +8,9 @@ class TestAction(ActionBase): self._logSys.info("ban ainfo %s, %s, %s, %s", aInfo["ipmatches"] != '', aInfo["ipjailmatches"] != '', aInfo["ipfailures"] > 0, aInfo["ipjailfailures"] > 0 ) + self._logSys.info("jail info %d, %d, %d, %d", + aInfo["jail.banned"], aInfo["jail.banned_total"], aInfo["jail.found"], aInfo["jail.found_total"] + ) def unban(self, aInfo): pass diff --git a/fail2ban/tests/files/logs/asterisk b/fail2ban/tests/files/logs/asterisk index 76ec40b2..ab31fa6f 100644 --- a/fail2ban/tests/files/logs/asterisk +++ b/fail2ban/tests/files/logs/asterisk @@ -19,6 +19,8 @@ [2012-02-13 17:44:26] NOTICE[1638] chan_iax2.c: Host 1.2.3.4 failed MD5 authentication for 'Fail2ban' (e7df7cd2ca07f4f1ab415d457a6e1c13 != 53ac4bc41ee4ec77888ed4aa50677247) # failJSON: { "time": "2013-02-05T23:44:42", "match": true , "host": "1.2.3.4" } [2013-02-05 23:44:42] NOTICE[436][C-00000fa9] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '0972598285108' rejected because extension not found in context 'default'. +# failJSON: { "time": "2005-01-18T17:39:50", "match": true , "host": "1.2.3.4" } +[Jan 18 17:39:50] NOTICE[12049]: res_pjsip_session.c:2337 new_invite: Call from 'anonymous' (TCP:[1.2.3.4]:61470) to extension '9011+442037690237' rejected because extension not found in context 'default'. # failJSON: { "time": "2013-03-26T15:47:54", "match": true , "host": "1.2.3.4" } [2013-03-26 15:47:54] NOTICE[1237] chan_sip.c: Registration from '"100"sip:100@1.2.3.4' failed for '1.2.3.4:23930' - No matching peer found # failJSON: { "time": "2013-05-13T07:10:53", "match": true , "host": "1.2.3.4" } diff --git a/fail2ban/tests/files/logs/drupal-auth b/fail2ban/tests/files/logs/drupal-auth index 5e7194d9..4d063e55 100644 --- a/fail2ban/tests/files/logs/drupal-auth +++ b/fail2ban/tests/files/logs/drupal-auth @@ -3,5 +3,15 @@ Apr 26 13:15:25 webserver example.com: https://example.com|1430068525|user|1.2.3 # failJSON: { "time": "2005-04-26T13:15:25", "match": true , "host": "1.2.3.4" } Apr 26 13:15:25 webserver example.com: https://example.com/subdir|1430068525|user|1.2.3.4|https://example.com/subdir/user|https://example.com/subdir/user|0||Login attempt failed for drupaladmin. -# failJSON: { "time": "2005-04-26T13:19:08", "match": false , "host": "1.2.3.4" } +# failJSON: { "time": "2005-04-26T13:19:08", "match": false , "host": "1.2.3.4", "user": "drupaladmin" } Apr 26 13:19:08 webserver example.com: https://example.com|1430068748|user|1.2.3.4|https://example.com/user|https://example.com/user|1||Session opened for drupaladmin. + +# failJSON: { "time": "2005-04-26T13:20:00", "match": false, "desc": "attempt to inject on URI (pipe, login failed for), not a failure, gh-2742" } +Apr 26 13:20:00 host drupal-site: https://example.com|1613063581|user|192.0.2.5|https://example.com/user/login?test=%7C&test2=%7C...|https://example.com/user/login?test=|&test2=|0||Login attempt failed for tester|2||Session revisited for drupaladmin. + +# failJSON: { "time": "2005-04-26T13:20:01", "match": true , "host": "192.0.2.7", "user": "Jack Sparrow", "desc": "log-format change - for -> from, user name with space, gh-2742" } +Apr 26 13:20:01 mweb drupal_site[24864]: https://www.example.com|1613058599|user|192.0.2.7|https://www.example.com/en/user/login|https://www.example.com/en/user/login|0||Login attempt failed from Jack Sparrow. +# failJSON: { "time": "2005-04-26T13:20:02", "match": true , "host": "192.0.2.4", "desc": "attempt to inject on URI (pipe), login failed, gh-2742" } +Apr 26 13:20:02 host drupal-site: https://example.com|1613063581|user|192.0.2.4|https://example.com/user/login?test=%7C&test2=%7C|https://example.com/user/login?test=|&test2=||0||Login attempt failed from 192.0.2.4. +# failJSON: { "time": "2005-04-26T13:20:03", "match": false, "desc": "attempt to inject on URI (pipe, login failed from), not a failure, gh-2742" } +Apr 26 13:20:03 host drupal-site: https://example.com|1613063581|user|192.0.2.5|https://example.com/user/login?test=%7C&test2=%7C...|https://example.com/user/login?test=|&test2=|0||Login attempt failed from 1.2.3.4|2||Session revisited for drupaladmin. diff --git a/fail2ban/tests/files/logs/monit b/fail2ban/tests/files/logs/monit index 8dbddaf6..36f1c1e4 100644 --- a/fail2ban/tests/files/logs/monit +++ b/fail2ban/tests/files/logs/monit @@ -1,7 +1,7 @@ # Previous version -- -# failJSON: { "time": "2005-04-16T21:05:29", "match": true , "host": "69.93.127.111" } +# failJSON: { "time": "2005-04-17T06:05:29", "match": true , "host": "69.93.127.111" } [PDT Apr 16 21:05:29] error : Warning: Client '69.93.127.111' supplied unknown user 'foo' accessing monit httpd -# failJSON: { "time": "2005-04-16T20:59:33", "match": true , "host": "97.113.189.111" } +# failJSON: { "time": "2005-04-17T05:59:33", "match": true , "host": "97.113.189.111" } [PDT Apr 16 20:59:33] error : Warning: Client '97.113.189.111' supplied wrong password for user 'admin' accessing monit httpd # Current version -- corresponding "https://bitbucket.org/tildeslash/monit/src/6905335aa903d425cae732cab766bd88ea5f2d1d/src/http/processor.c?at=master&fileviewer=file-view-default#processor.c-728" diff --git a/fail2ban/tests/files/logs/monitorix b/fail2ban/tests/files/logs/monitorix new file mode 100644 index 00000000..e6ad6dc6 --- /dev/null +++ b/fail2ban/tests/files/logs/monitorix @@ -0,0 +1,8 @@ +# failJSON: { "time": "2021-04-14T08:11:01", "match": false, "desc": "should be ignored: successful request" } +Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix-cgi/monitorix.cgi - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" +# failJSON: { "time": "2021-04-14T08:54:22", "match": true, "host": "127.0.0.1", "desc": "file does not exist" } +Wed Apr 14 08:54:22 2021 - NOTEXIST - [127.0.0.1] File does not exist: /manager/html +# failJSON: { "time": "2021-04-14T11:24:31", "match": true, "host": "127.0.0.1", "desc": "access not allowed" } +Wed Apr 14 11:24:31 2021 - NOTALLOWED - [127.0.0.1] Access not allowed: /monitorix/ +# failJSON: { "time": "2021-04-14T11:26:08", "match": true, "host": "127.0.0.1", "desc": "authentication error" } +Wed Apr 14 11:26:08 2021 - AUTHERR - [127.0.0.1] Authentication error: /monitorix/ diff --git a/fail2ban/tests/files/logs/mssql-auth b/fail2ban/tests/files/logs/mssql-auth new file mode 100644 index 00000000..1c9b65ec --- /dev/null +++ b/fail2ban/tests/files/logs/mssql-auth @@ -0,0 +1,11 @@ +# failJSON: { "time": "2020-02-24T16:05:21", "match": true , "host": "192.0.2.1" } +2020-02-24 16:05:21.00 Logon Login failed for user 'Backend'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.1] +# failJSON: { "time": "2020-02-24T16:30:25", "match": true , "host": "192.0.2.2" } +2020-02-24 16:30:25.88 Logon Login failed for user '===)jf02hüas9ä##22f'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.2] +# failJSON: { "time": "2020-02-24T16:31:12", "match": true , "host": "192.0.2.3" } +2020-02-24 16:31:12.20 Logon Login failed for user ''. Reason: An attempt to login using SQL authentication failed. Server is configured for Integrated authentication only. [CLIENT: 192.0.2.3] + +# failJSON: { "time": "2020-02-24T16:31:26", "match": true , "host": "192.0.2.4", "user":"O'Leary" } +2020-02-24 16:31:26.01 Logon Login failed for user 'O'Leary'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.4] +# failJSON: { "time": "2020-02-24T16:31:26", "match": false, "desc": "test injection in possibly unescaped foreign input" } +2020-02-24 16:31:26.02 Wrong data received: Logon Login failed for user 'test'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.5] diff --git a/fail2ban/tests/files/logs/nginx-bad-request b/fail2ban/tests/files/logs/nginx-bad-request new file mode 100644 index 00000000..a9ff6497 --- /dev/null +++ b/fail2ban/tests/files/logs/nginx-bad-request @@ -0,0 +1,23 @@ +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - root [20/Jan/2015:19:53:28 +0100] "" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "GET //admin/pma/scripts/setup.php HTTP/1.1" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:54:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:54:28 +0100] "HELP" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T19:55:28", "match": true , "host": "12.34.56.78" } +12.34.56.78 - - [20/Jan/2015:19:55:28 +0100] "batman" 400 47 "-" "-" "-" + +# failJSON: { "time": "2015-01-20T01:17:07", "match": true , "host": "7.8.9.10" } +7.8.9.10 - root [20/Jan/2015:01:17:07 +0100] "CONNECT 123.123.123.123 HTTP/1.1" 400 162 "-" "-" "-" + +# failJSON: { "time": "2014-12-12T22:59:02", "match": true , "host": "2.5.2.5" } +2.5.2.5 - tomcat [12/Dec/2014:22:59:02 +0100] "GET /cgi-bin/tools/tools.pl HTTP/1.1" 400 162 "-" "-" "-"
\ No newline at end of file diff --git a/fail2ban/tests/files/logs/nginx-http-auth b/fail2ban/tests/files/logs/nginx-http-auth index c9c96807..fb24b242 100644 --- a/fail2ban/tests/files/logs/nginx-http-auth +++ b/fail2ban/tests/files/logs/nginx-http-auth @@ -1,3 +1,4 @@ +# filterOptions: [{"mode": "normal"}, {"mode": "auth"}] # failJSON: { "time": "2012-04-09T11:53:29", "match": true , "host": "192.0.43.10" } 2012/04/09 11:53:29 [error] 2865#0: *66647 user "xyz" was not found in "/var/www/.htpasswd", client: 192.0.43.10, server: www.myhost.com, request: "GET / HTTP/1.1", host: "www.myhost.com" @@ -11,3 +12,20 @@ 2014/04/03 22:20:38 [error] 30708#0: *3 user "scriben dio": password mismatch, client: 192.0.2.1, server: , request: "GET / HTTP/1.1", host: "localhost:8443" # failJSON: { "time": "2014-04-03T22:20:40", "match": true, "host": "192.0.2.2", "desc": "trying injection on user name"} 2014/04/03 22:20:40 [error] 30708#0: *3 user "test": password mismatch, client: 127.0.0.1, server: test, request: "GET / HTTP/1.1", host: "localhost:8443"": was not found in "/etc/nginx/.htpasswd", client: 192.0.2.2, server: , request: "GET / HTTP/1.1", host: "localhost:8443" + +# filterOptions: [{"mode": "fallback"}] + +# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" } +2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T15:47:47", "match": true , "host": "80.191.166.166" } +2020/11/25 15:47:47 [crit] 76952#76952: *5062354 SSL_do_handshake() failed (SSL: error:1408F0A0:SSL routines:ssl3_get_record:length too short) while SSL handshaking, client: 80.191.166.166, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T16:48:08", "match": true , "host": "5.126.32.148" } +2020/11/25 16:48:08 [crit] 76952#76952: *7976400 SSL_do_handshake() failed (SSL: error:1408F096:SSL routines:ssl3_get_record:encrypted length too long) while SSL handshaking, client: 5.126.32.148, server: 0.0.0.0:443 +# failJSON: { "time": "2020-11-25T16:02:45", "match": false } +2020/11/25 16:02:45 [error] 76952#76952: *5645766 connect() failed (111: Connection refused) while connecting to upstream, client: 5.126.32.148, server: www.google.de, request: "GET /admin/config HTTP/2.0", upstream: "http://127.0.0.1:3000/admin/config", host: "www.google.de" + +# filterOptions: [{"mode": "aggressive"}] +# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" } +2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443 +# failJSON: { "time": "2012-04-09T11:53:29", "match": true , "host": "192.0.43.10" } +2012/04/09 11:53:29 [error] 2865#0: *66647 user "xyz" was not found in "/var/www/.htpasswd", client: 192.0.43.10, server: www.myhost.com, request: "GET / HTTP/1.1", host: "www.myhost.com" diff --git a/fail2ban/tests/files/logs/nsd b/fail2ban/tests/files/logs/nsd index a33a52a9..63c162e9 100644 --- a/fail2ban/tests/files/logs/nsd +++ b/fail2ban/tests/files/logs/nsd @@ -2,3 +2,5 @@ [1387288694] nsd[7745]: info: ratelimit block example.com. type any target 192.0.2.0/24 query 192.0.2.105 TYPE255 # failJSON: { "time": "2013-12-18T07:42:15", "match": true , "host": "192.0.2.115" } [1387348935] nsd[23600]: info: axfr for zone domain.nl. from client 192.0.2.115 refused, no acl matches. +# failJSON: { "time": "2021-03-05T05:25:14", "match": true , "host": "192.0.2.32", "desc": "new format, no client after from, no dot at end, gh-2965" } +[2021-03-05 05:25:14.562] nsd[160800]: info: axfr for example.com. from 192.0.2.32 refused, no acl matches diff --git a/fail2ban/tests/files/logs/postfix b/fail2ban/tests/files/logs/postfix index 85b61ea6..d1e534e3 100644 --- a/fail2ban/tests/files/logs/postfix +++ b/fail2ban/tests/files/logs/postfix @@ -15,6 +15,9 @@ Aug 10 10:55:38 f-vanier-bourgeois postfix/smtpd[2162]: NOQUEUE: reject: VRFY fr # failJSON: { "time": "2005-08-13T15:45:46", "match": true , "host": "192.0.2.1" } Aug 13 15:45:46 server postfix/smtpd[13844]: 00ADB3C0899: reject: RCPT from example.com[192.0.2.1]: 550 5.1.1 <sales@server.com>: Recipient address rejected: User unknown in local recipient table; from=<xxxxxx@example.com> to=<sales@server.com> proto=ESMTP helo=<mail.example.com> +# failJSON: { "time": "2005-05-19T00:00:30", "match": true , "host": "192.0.2.2", "desc": "undeliverable address (sender/recipient verification, gh-3039)" } +May 19 00:00:30 proxy2 postfix/smtpd[16123]: NOQUEUE: reject: RCPT from example.net[192.0.2.2]: 550 5.1.1 <user1@example.com>: Recipient address rejected: undeliverable address: verification failed; from=<user2@example.org> to=<user1@example.com> proto=ESMTP helo=<example.net> + # failJSON: { "time": "2005-01-12T11:07:49", "match": true , "host": "181.21.131.88" } Jan 12 11:07:49 emf1pt2-2-35-70 postfix/smtpd[13767]: improper command pipelining after DATA from unknown[181.21.131.88]: @@ -161,6 +164,17 @@ Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[1 # failJSON: { "time": "2005-02-18T09:48:04", "match": true , "host": "192.0.2.23" } Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[192.0.2.23] +# failJSON: { "time": "2004-12-23T19:39:13", "match": true , "host": "192.0.2.2" } +Dec 23 19:39:13 xxx postfix/postscreen[21057]: PREGREET 14 after 0.08 from [192.0.2.2]:59415: EHLO ylmf-pc\r\n +# failJSON: { "time": "2004-12-24T00:54:36", "match": true , "host": "192.0.2.3" } +Dec 24 00:54:36 xxx postfix/postscreen[22515]: HANGUP after 16 from [192.0.2.3]:48119 in tests after SMTP handshake + +# failJSON: { "time": "2005-06-08T23:14:28", "match": true , "host": "192.0.2.77", "desc": "abusive clients hitting command limit, see see http://www.postfix.org/POSTSCREEN_README.html (gh-3040)" } +Jun 8 23:14:28 proxy2 postfix/postscreen[473]: COMMAND TIME LIMIT from [192.0.2.77]:3608 after CONNECT +# failJSON: { "time": "2005-06-08T23:14:54", "match": true , "host": "192.0.2.26", "desc": "abusive clients hitting command limit (gh-3040)" } +Jun 8 23:14:54 proxy2 postfix/postscreen[473]: COMMAND COUNT LIMIT from [192.0.2.26]:15592 after RCPT + + # filterOptions: [{}, {"mode": "ddos"}, {"mode": "aggressive"}] # failJSON: { "match": false, "desc": "don't affect lawful data (sporadical connection aborts within DATA-phase, see gh-1813 for discussion)" } Feb 18 09:50:05 xxx postfix/smtpd[42]: lost connection after DATA from good-host.example.com[192.0.2.10] diff --git a/fail2ban/tests/files/logs/scanlogd b/fail2ban/tests/files/logs/scanlogd new file mode 100644 index 00000000..5a97c578 --- /dev/null +++ b/fail2ban/tests/files/logs/scanlogd @@ -0,0 +1,8 @@ +# failJSON: { "time": "2005-03-05T21:44:43", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:43 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 80, 81, 83, 88, 99, 443, 1080, 3128, ..., f????uxy, TOS 00, TTL 49 @20:44:43 +# failJSON: { "time": "2005-03-05T21:44:44", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:44 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 497, 515, 544, 543, 464, 513, ..., fSrpauxy, TOS 00 @09:04:25 +# failJSON: { "time": "2005-03-05T21:44:45", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:45 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 593, 548, 636, 646, 625, 631, ..., fSrpauxy, TOS 00, TTL 239 @17:34:00 +# failJSON: { "time": "2005-03-05T21:44:46", "match": true , "host": "192.0.2.123" } +Mar 5 21:44:46 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 22, 26, 37, 80, 25, 79, ..., fSrpauxy, TOS 00 @22:38:37 diff --git a/fail2ban/tests/files/logs/zoneminder b/fail2ban/tests/files/logs/zoneminder index abd49869..f4b6bd3e 100644 --- a/fail2ban/tests/files/logs/zoneminder +++ b/fail2ban/tests/files/logs/zoneminder @@ -1,2 +1,8 @@ # failJSON: { "time": "2016-03-28T16:50:49", "match": true , "host": "10.1.1.1" } [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/ + +# failJSON: { "time": "2021-03-28T16:53:00", "match": true , "host": "10.1.1.1" } +[Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user username1 details], referer: https://zm/zm/?view=logout + +# failJSON: { "time": "2021-03-28T16:59:14", "match": true , "host": "10.1.1.1" } +[Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "username1"], referer: https://zm/zm/? diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index a89b8364..017e54ec 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -141,7 +141,7 @@ def _ticket_tuple(ticket): """ attempts = ticket.getAttempt() date = ticket.getTime() - ip = ticket.getIP() + ip = ticket.getID() matches = ticket.getMatches() return (ip, attempts, date, matches) @@ -196,7 +196,7 @@ def _assert_correct_last_attempt(utest, filter_, output, count=None): _assert_equal_entries(utest, f, o) -def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line=""): +def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line="", lines=None): """Copy lines from one file to another (which might be already open) Returns open fout @@ -213,9 +213,9 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line fin.readline() # Read i = 0 - lines = [] + if not lines: lines = [] while n is None or i < n: - l = FileContainer.decode_line(in_, 'UTF-8', fin.readline()).rstrip('\r\n') + l = fin.readline().decode('UTF-8', 'replace').rstrip('\r\n') if terminal_line is not None and l == terminal_line: break lines.append(l) @@ -223,6 +223,7 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line # Write: all at once and flush if isinstance(fout, str): fout = open(fout, mode) + DefLogSys.debug(' ++ write %d test lines', len(lines)) fout.write('\n'.join(lines)+'\n') fout.flush() if isinstance(in_, str): # pragma: no branch - only used with str in test cases @@ -254,7 +255,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p # Read/Write i = 0 while n is None or i < n: - l = FileContainer.decode_line(in_, 'UTF-8', fin.readline()).rstrip('\r\n') + l = fin.readline().decode('UTF-8', 'replace').rstrip('\r\n') if terminal_line is not None and l == terminal_line: break journal.send(MESSAGE=l.strip(), **fields) @@ -670,6 +671,19 @@ class LogFile(LogCaptureTestCase): self.filter = FilterPoll(None) self.assertRaises(IOError, self.filter.addLogPath, LogFile.MISSING) + def testDecodeLineWarn(self): + # incomplete line (missing byte at end), warning is suppressed: + l = u"correct line\n" + r = l.encode('utf-16le') + self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-16le', r), l) + self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-16le', r[0:-1]), l[0:-1]) + self.assertNotLogged('Error decoding line') + # complete line (incorrect surrogate in the middle), warning is there: + r = b"incorrect \xc8\x0a line\n" + l = r.decode('utf-8', 'replace') + self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-8', r), l) + self.assertLogged('Error decoding line') + class LogFileFilterPoll(unittest.TestCase): @@ -1179,13 +1193,15 @@ def get_monitor_failures_testcase(Filter_): # move aside, but leaving the handle still open... os.rename(self.name, self.name + '.bak') - _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1).close() + _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1, + lines=["Aug 14 11:59:59 [logrotate] rotation 1"]).close() self.assert_correct_last_attempt(GetFailures.FAILURES_01) self.assertEqual(self.filter.failManager.getFailTotal(), 3) # now remove the moved file _killfile(None, self.name + '.bak') - _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=12, n=3).close() + _copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=12, n=3, + lines=["Aug 14 11:59:59 [logrotate] rotation 2"]).close() self.assert_correct_last_attempt(GetFailures.FAILURES_01) self.assertEqual(self.filter.failManager.getFailTotal(), 6) @@ -1239,7 +1255,7 @@ def get_monitor_failures_testcase(Filter_): os.rename(tmpsub1, tmpsub2 + 'a') os.mkdir(tmpsub1) self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, - skip=12, n=1, mode='w') + skip=12, n=1, mode='w', lines=["Aug 14 11:59:59 [logrotate] rotation 1"]) self.file.close() self._wait4failures(2) @@ -1250,7 +1266,7 @@ def get_monitor_failures_testcase(Filter_): os.mkdir(tmpsub1) self.waitForTicks(2) self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name, - skip=12, n=1, mode='w') + skip=12, n=1, mode='w', lines=["Aug 14 11:59:59 [logrotate] rotation 2"]) self.file.close() self._wait4failures(3) @@ -1444,7 +1460,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.assertTrue(ticket) attempts = ticket.getAttempt() - ip = ticket.getIP() + ip = ticket.getID() ticket.getMatches() self.assertEqual(ip, test_ip) @@ -1630,7 +1646,7 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover self.waitForTicks(1) self.waitFailTotal(6, 10) self.assertTrue(Utils.wait_for(lambda: len(self.jail) == 2, 10)) - self.assertSortedEqual([self.jail.getFailTicket().getIP(), self.jail.getFailTicket().getIP()], + self.assertSortedEqual([self.jail.getFailTicket().getID(), self.jail.getFailTicket().getID()], ["192.0.2.1", "192.0.2.2"]) cls = MonitorJournalFailures @@ -1712,16 +1728,49 @@ class GetFailures(LogCaptureTestCase): def testCRLFFailures01(self): # We first adjust logfile/failures to end with CR+LF fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='crlf') - # poor man unix2dos: - fin, fout = open(GetFailures.FILENAME_01, 'rb'), open(fname, 'wb') - for l in fin.read().splitlines(): - fout.write(l + b'\r\n') - fin.close() - fout.close() + try: + # poor man unix2dos: + fin, fout = open(GetFailures.FILENAME_01, 'rb'), open(fname, 'wb') + for l in fin.read().splitlines(): + fout.write(l + b'\r\n') + fin.close() + fout.close() - # now see if we should be getting the "same" failures - self.testGetFailures01(filename=fname) - _killfile(fout, fname) + # now see if we should be getting the "same" failures + self.testGetFailures01(filename=fname) + finally: + _killfile(fout, fname) + + def testNLCharAsPartOfUniChar(self): + fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni') + # test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02): + for enc in ('utf-16be', 'utf-16le'): + self.pruneLog("[test-phase encoding=%s]" % enc) + try: + fout = open(fname, 'wb') + tm = int(time.time()) + # test on unicode string containing \x0A as part of uni-char, + # it must produce exactly 2 lines (both are failures): + for l in ( + u'%s \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n' % tm, + u'%s \u20AC Failed auth: invalid user TestI from 192.0.2.2\n' % tm + ): + fout.write(l.encode(enc)) + fout.close() + + self.filter.setLogEncoding(enc) + self.filter.addLogPath(fname, autoSeek=0) + self.filter.setDatePattern((r'^EPOCH',)) + self.filter.addFailRegex(r"Failed .* from <HOST>") + self.filter.getFailures(fname) + self.assertLogged( + "[DummyJail] Found 192.0.2.1", + "[DummyJail] Found 192.0.2.2", all=True, wait=True) + finally: + _killfile(fout, fname) + self.filter.delLogPath(fname) + # must find 4 failures and generate 2 tickets (2 IPs with each 2 failures): + self.assertEqual(self.filter.failManager.getFailCount(), (2, 4)) def testGetFailures02(self): output = ('141.3.81.106', 4, 1124013539.0, @@ -2285,6 +2334,7 @@ class DNSUtilsNetworkTests(unittest.TestCase): ip1 = IPAddr('2606:2800:220:1:248:1893:25c8:1946'); ip2 = IPAddr('2606:2800:220:1:248:1893:25c8:1946'); self.assertEqual(id(ip1), id(ip2)) def testFQDN(self): + unittest.F2B.SkipIfNoNetwork() sname = DNSUtils.getHostname(fqdn=False) lname = DNSUtils.getHostname(fqdn=True) # FQDN is not localhost if short hostname is not localhost too (or vice versa): diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index e2faa6fd..4b026377 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -70,16 +70,10 @@ class HelpersTest(unittest.TestCase): self.assertEqual(splitwords(u' 1\n 2, 3'), ['1', '2', '3']) -if sys.version_info >= (2,7): - def _sh_call(cmd): - import subprocess - ret = subprocess.check_output(cmd, shell=True) - return uni_decode(ret).rstrip() -else: - def _sh_call(cmd): - import subprocess - ret = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read() - return uni_decode(ret).rstrip() +def _sh_call(cmd): + import subprocess + ret = subprocess.check_output(cmd, shell=True) + return uni_decode(ret).rstrip() def _getSysPythonVersion(): return _sh_call("fail2ban-python -c 'import sys; print(tuple(sys.version_info))'") @@ -92,7 +86,7 @@ class SetupTest(unittest.TestCase): unittest.F2B.SkipIfFast() setup = os.path.join(os.path.dirname(__file__), '..', '..', 'setup.py') self.setup = os.path.exists(setup) and setup or None - if not self.setup and sys.version_info >= (2,7): # pragma: no cover - running not out of the source + if not self.setup: # pragma: no cover - running not out of the source raise unittest.SkipTest( "Seems to be running not out of source distribution" " -- cannot locate setup.py") diff --git a/fail2ban/tests/observertestcase.py b/fail2ban/tests/observertestcase.py index e379ccd1..9b44c6dd 100644 --- a/fail2ban/tests/observertestcase.py +++ b/fail2ban/tests/observertestcase.py @@ -181,7 +181,7 @@ class BanTimeIncrDB(LogCaptureTestCase): def setUp(self): """Call before every test case.""" super(BanTimeIncrDB, self).setUp() - if Fail2BanDb is None and sys.version_info >= (2,7): # pragma: no cover + if Fail2BanDb is None: # pragma: no cover raise unittest.SkipTest( "Unable to import fail2ban database module as sqlite is not " "available.") @@ -367,14 +367,14 @@ class BanTimeIncrDB(LogCaptureTestCase): # this old ticket should be removed now: restored_tickets = self.db.getCurrentBans(fromtime=stime, correctBanTime=False) self.assertEqual(len(restored_tickets), 2) - self.assertEqual(restored_tickets[0].getIP(), ip) + self.assertEqual(restored_tickets[0].getID(), ip) # purge remove 1st ip self.db._purgeAge = -48*60*60 self.db.purge() restored_tickets = self.db.getCurrentBans(fromtime=stime, correctBanTime=False) self.assertEqual(len(restored_tickets), 1) - self.assertEqual(restored_tickets[0].getIP(), ip+'1') + self.assertEqual(restored_tickets[0].getID(), ip+'1') # this should purge all bans, bips and logs - nothing should be found now self.db._purgeAge = -240*60*60 @@ -450,7 +450,8 @@ class BanTimeIncrDB(LogCaptureTestCase): def testObserver(self): if Fail2BanDb is None: # pragma: no cover return - jail = self.jail + jail = self.jail = DummyJail(backend='polling') + jail.database = self.db self.db.addJail(jail) # we tests with initial ban time = 10 seconds: jail.actions.setBanTime(10) @@ -480,27 +481,27 @@ class BanTimeIncrDB(LogCaptureTestCase): # add failure: ip = "192.0.2.1" ticket = FailTicket(ip, stime-120, []) - failManager = FailManager() + failManager = jail.filter.failManager = FailManager() failManager.setMaxRetry(3) for i in xrange(3): failManager.addFailure(ticket) - obs.add('failureFound', failManager, jail, ticket) + obs.add('failureFound', jail, ticket) obs.wait_empty(5) self.assertEqual(ticket.getBanCount(), 0) # check still not ban : self.assertTrue(not jail.getFailTicket()) # add manually 4th times banned (added to bips - make ip bad): ticket.setBanCount(4) - self.db.addBan(self.jail, ticket) + self.db.addBan(jail, ticket) restored_tickets = self.db.getCurrentBans(jail=jail, fromtime=stime-120, correctBanTime=False) self.assertEqual(len(restored_tickets), 1) # check again, new ticket, new failmanager: ticket = FailTicket(ip, stime, []) - failManager = FailManager() + failManager = jail.filter.failManager = FailManager() failManager.setMaxRetry(3) # add once only - but bad - should be banned: failManager.addFailure(ticket) - obs.add('failureFound', failManager, self.jail, ticket) + obs.add('failureFound', jail, ticket) obs.wait_empty(5) # wait until ticket transfered from failmanager into jail: ticket2 = Utils.wait_for(jail.getFailTicket, 10) diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py index 5a72ffa9..b33b46c1 100644 --- a/fail2ban/tests/samplestestcase.py +++ b/fail2ban/tests/samplestestcase.py @@ -23,7 +23,6 @@ __copyright__ = "Copyright (c) 2013 Steven Hiscocks" __license__ = "GPL" import datetime -import fileinput import inspect import json import os @@ -156,12 +155,15 @@ def testSampleRegexsFactory(name, basedir): i = 0 while i < len(filenames): filename = filenames[i]; i += 1; - logFile = fileinput.FileInput(os.path.join(TEST_FILES_DIR, "logs", - filename), mode='rb') + logFile = FileContainer(os.path.join(TEST_FILES_DIR, "logs", + filename), 'UTF-8', doOpen=True) + # avoid errors if no NL char at end of test log-file: + logFile.waitForLineEnd = False ignoreBlock = False + lnnum = 0 for line in logFile: - line = FileContainer.decode_line(logFile.filename(), 'UTF-8', line) + lnnum += 1 jsonREMatch = re.match("^#+ ?(failJSON|(?:file|filter)Options|addFILE):(.+)$", line) if jsonREMatch: try: @@ -201,9 +203,8 @@ def testSampleRegexsFactory(name, basedir): # failJSON - faildata contains info of the failure to check it. except ValueError as e: # pragma: no cover - we've valid json's raise ValueError("%s: %s:%i" % - (e, logFile.filename(), logFile.filelineno())) + (e, logFile.getFileName(), lnnum)) line = next(logFile) - line = FileContainer.decode_line(logFile.filename(), 'UTF-8', line) elif ignoreBlock or line.startswith("#") or not line.strip(): continue else: # pragma: no cover - normally unreachable @@ -298,7 +299,7 @@ def testSampleRegexsFactory(name, basedir): import pprint raise AssertionError("%s: %s on: %s:%i, line:\n %s\nregex (%s):\n %s\n" "faildata: %s\nfail: %s" % ( - fltName, e, logFile.filename(), logFile.filelineno(), + fltName, e, logFile.getFileName(), lnnum, line, failregex, regexList[failregex] if failregex != -1 else None, '\n'.join(pprint.pformat(faildata).splitlines()), '\n'.join(pprint.pformat(fail).splitlines()))) diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 7a0685eb..62ae81fd 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -775,27 +775,11 @@ class Transmitter(TransmitterBase): def testPythonActionMethodsAndProperties(self): action = "TestCaseAction" - try: - out = self.transm.proceed( - ["set", self.jailName, "addaction", action, - os.path.join(TEST_FILES_DIR, "action.d", "action.py"), - '{"opt1": "value"}']) - self.assertEqual(out, (0, action)) - except AssertionError: - if ((2, 6) <= sys.version_info < (2, 6, 5)) \ - and '__init__() keywords must be strings' in out[1]: - # known issue http://bugs.python.org/issue2646 in 2.6 series - # since general Fail2Ban warnings are suppressed in normal - # operation -- let's issue Python's native warning here - import warnings - warnings.warn( - "Your version of Python %s seems to experience a known " - "issue forbidding correct operation of Fail2Ban: " - "http://bugs.python.org/issue2646 Upgrade your Python and " - "meanwhile other intestPythonActionMethodsAndProperties will " - "be skipped" % (sys.version)) - return - raise + out = self.transm.proceed( + ["set", self.jailName, "addaction", action, + os.path.join(TEST_FILES_DIR, "action.d", "action.py"), + '{"opt1": "value"}']) + self.assertEqual(out, (0, action)) self.assertSortedEqual( self.transm.proceed(["get", self.jailName, "actionproperties", action])[1], @@ -1503,35 +1487,42 @@ class ServerConfigReaderTests(LogCaptureTestCase): ), }), # iptables-multiport -- - ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp", chain="<known/chain>"]', { + ('j-w-iptables-mp', 'iptables-multiport[name=%(__name__)s, bantime="10m", port="http,https", protocol="tcp,udp,sctp", chain="<known/chain>"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + r"`for proto in $(echo 'tcp,udp,sctp' | sed 's/,/ /g'); do`", + r"`done`", + ), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-mp`", - "`iptables -w -A f2b-j-w-iptables-mp -j RETURN`", - "`iptables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`{ iptables -w -C f2b-j-w-iptables-mp -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables-mp || true; iptables -w -A f2b-j-w-iptables-mp -j RETURN; }`", + "`{ iptables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-mp`", - "`ip6tables -w -A f2b-j-w-iptables-mp -j RETURN`", - "`ip6tables -w -I INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`{ ip6tables -w -C f2b-j-w-iptables-mp -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables-mp || true; ip6tables -w -A f2b-j-w-iptables-mp -j RETURN; }`", + "`{ ip6tables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp >/dev/null 2>&1; } || ", + "{ ip6tables -w -I INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-mp`", "`ip6tables -w -F f2b-j-w-iptables-mp`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`iptables -w -D INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`", "`iptables -w -F f2b-j-w-iptables-mp`", "`iptables -w -X f2b-j-w-iptables-mp`", - "`ip6tables -w -D INPUT -p tcp -m multiport --dports http,https -j f2b-j-w-iptables-mp`", + "`ip6tables -w -D INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`", "`ip6tables -w -F f2b-j-w-iptables-mp`", "`ip6tables -w -X f2b-j-w-iptables-mp`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", + r"""`iptables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-mp[ \t]'`""", + r"""`ip6tables -w -C INPUT -p $proto -m multiport --dports http,https -j f2b-j-w-iptables-mp`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-mp 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1547,35 +1538,42 @@ class ServerConfigReaderTests(LogCaptureTestCase): ), }), # iptables-allports -- - ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp", chain="<known/chain>"]', { + ('j-w-iptables-ap', 'iptables-allports[name=%(__name__)s, bantime="10m", protocol="tcp,udp,sctp", chain="<known/chain>"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + r"`for proto in $(echo 'tcp,udp,sctp' | sed 's/,/ /g'); do`", + r"`done`", + ), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-ap`", - "`iptables -w -A f2b-j-w-iptables-ap -j RETURN`", - "`iptables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`{ iptables -w -C f2b-j-w-iptables-ap -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables-ap || true; iptables -w -A f2b-j-w-iptables-ap -j RETURN; }`", + "`{ iptables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", + "{ iptables -w -I INPUT -p $proto -j f2b-j-w-iptables-ap; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-ap`", - "`ip6tables -w -A f2b-j-w-iptables-ap -j RETURN`", - "`ip6tables -w -I INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`{ ip6tables -w -C f2b-j-w-iptables-ap -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables-ap || true; ip6tables -w -A f2b-j-w-iptables-ap -j RETURN; }`", + "`{ ip6tables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap >/dev/null 2>&1; } || ", + "{ ip6tables -w -I INPUT -p $proto -j f2b-j-w-iptables-ap; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-ap`", "`ip6tables -w -F f2b-j-w-iptables-ap`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`iptables -w -D INPUT -p $proto -j f2b-j-w-iptables-ap`", "`iptables -w -F f2b-j-w-iptables-ap`", "`iptables -w -X f2b-j-w-iptables-ap`", - "`ip6tables -w -D INPUT -p tcp -j f2b-j-w-iptables-ap`", + "`ip6tables -w -D INPUT -p $proto -j f2b-j-w-iptables-ap`", "`ip6tables -w -F f2b-j-w-iptables-ap`", "`ip6tables -w -X f2b-j-w-iptables-ap`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", + r"""`iptables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-ap[ \t]'`""", + r"""`ip6tables -w -C INPUT -p $proto -j f2b-j-w-iptables-ap`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-ap 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1593,105 +1591,138 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-ipset-proto6 -- ('j-w-iptables-ipset', 'iptables-ipset-proto6[name=%(__name__)s, port="http", protocol="tcp", chain="<known/chain>"]', { 'ip4': (' f2b-j-w-iptables-ipset ',), 'ip6': (' f2b-j-w-iptables-ipset6 ',), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( - "`ipset create f2b-j-w-iptables-ipset hash:ip timeout 0 `", - "`iptables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + "`ipset -exist create f2b-j-w-iptables-ipset hash:ip timeout 0 `", + "`{ iptables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable; }`", ), 'ip6-start': ( - "`ipset create f2b-j-w-iptables-ipset6 hash:ip timeout 0 family inet6`", - "`ip6tables -w -I INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ipset -exist create f2b-j-w-iptables-ipset6 hash:ip timeout 0 family inet6`", + "`{ ip6tables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable; }`", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset`", "`ipset flush f2b-j-w-iptables-ipset6`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", + "`iptables -w -D INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset`", "`ipset destroy f2b-j-w-iptables-ipset`", - "`ip6tables -w -D INPUT -p tcp -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ip6tables -w -D INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset6`", "`ipset destroy f2b-j-w-iptables-ipset6`", ), + 'ip4-check': ( + r"""`iptables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset src -j REJECT --reject-with icmp-port-unreachable`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -C INPUT -p $proto -m multiport --dports http -m set --match-set f2b-j-w-iptables-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`""", + ), 'ip4-ban': ( - r"`ipset add f2b-j-w-iptables-ipset 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-iptables-ipset 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-iptables-ipset6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-iptables-ipset6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset6 2001:db8::`", ), }), # iptables-ipset-proto6-allports -- ('j-w-iptables-ipset-ap', 'iptables-ipset-proto6-allports[name=%(__name__)s, chain="<known/chain>"]', { 'ip4': (' f2b-j-w-iptables-ipset-ap ',), 'ip6': (' f2b-j-w-iptables-ipset-ap6 ',), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( - "`ipset create f2b-j-w-iptables-ipset-ap hash:ip timeout 0 `", - "`iptables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`ipset -exist create f2b-j-w-iptables-ipset-ap hash:ip timeout 0 `", + "`{ iptables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable; }", ), 'ip6-start': ( - "`ipset create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", - "`ip6tables -w -I INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ipset -exist create f2b-j-w-iptables-ipset-ap6 hash:ip timeout 0 family inet6`", + "`{ ip6tables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable; }", ), 'flush': ( "`ipset flush f2b-j-w-iptables-ipset-ap`", "`ipset flush f2b-j-w-iptables-ipset-ap6`", ), 'stop': ( - "`iptables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", + "`iptables -w -D INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap`", "`ipset destroy f2b-j-w-iptables-ipset-ap`", - "`ip6tables -w -D INPUT -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", + "`ip6tables -w -D INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", "`ipset flush f2b-j-w-iptables-ipset-ap6`", "`ipset destroy f2b-j-w-iptables-ipset-ap6`", ), + 'ip4-check': ( + r"""`iptables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`""", + ), + 'ip6-check': ( + r"""`ip6tables -w -C INPUT -p $proto -m set --match-set f2b-j-w-iptables-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`""", + ), 'ip4-ban': ( - r"`ipset add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset-ap 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-iptables-ipset-ap 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset-ap 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-iptables-ipset-ap6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-iptables-ipset-ap6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-iptables-ipset-ap6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-iptables-ipset-ap6 2001:db8::`", ), }), - # iptables -- + # iptables (oneport) -- ('j-w-iptables', 'iptables[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="<known/chain>"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables`", - "`iptables -w -A f2b-j-w-iptables -j RETURN`", - "`iptables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`{ iptables -w -C f2b-j-w-iptables -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables || true; iptables -w -A f2b-j-w-iptables -j RETURN; }", + "`{ iptables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -p $proto --dport http -j f2b-j-w-iptables; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables`", - "`ip6tables -w -A f2b-j-w-iptables -j RETURN`", - "`ip6tables -w -I INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`{ ip6tables -w -C f2b-j-w-iptables -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables || true; ip6tables -w -A f2b-j-w-iptables -j RETURN; }", + "`{ ip6tables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -p $proto --dport http -j f2b-j-w-iptables; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables`", "`ip6tables -w -F f2b-j-w-iptables`", ), 'stop': ( - "`iptables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`iptables -w -D INPUT -p $proto --dport http -j f2b-j-w-iptables`", "`iptables -w -F f2b-j-w-iptables`", "`iptables -w -X f2b-j-w-iptables`", - "`ip6tables -w -D INPUT -p tcp --dport http -j f2b-j-w-iptables`", + "`ip6tables -w -D INPUT -p $proto --dport http -j f2b-j-w-iptables`", "`ip6tables -w -F f2b-j-w-iptables`", "`ip6tables -w -X f2b-j-w-iptables`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", + r"""`iptables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables[ \t]'`""", + r"""`ip6tables -w -C INPUT -p $proto --dport http -j f2b-j-w-iptables`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1709,33 +1740,40 @@ class ServerConfigReaderTests(LogCaptureTestCase): # iptables-new -- ('j-w-iptables-new', 'iptables-new[name=%(__name__)s, bantime="10m", port="http", protocol="tcp", chain="<known/chain>"]', { 'ip4': ('`iptables ', 'icmp-port-unreachable'), 'ip6': ('`ip6tables ', 'icmp6-port-unreachable'), + '*-start-stop-check': ( + # iterator over protocol is same for both families: + "`for proto in $(echo 'tcp' | sed 's/,/ /g'); do`", + "`done`", + ), 'ip4-start': ( - "`iptables -w -N f2b-j-w-iptables-new`", - "`iptables -w -A f2b-j-w-iptables-new -j RETURN`", - "`iptables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`{ iptables -w -C f2b-j-w-iptables-new -j RETURN >/dev/null 2>&1; } || " + "{ iptables -w -N f2b-j-w-iptables-new || true; iptables -w -A f2b-j-w-iptables-new -j RETURN; }`", + "`{ iptables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " + "{ iptables -w -I INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new; }`", ), 'ip6-start': ( - "`ip6tables -w -N f2b-j-w-iptables-new`", - "`ip6tables -w -A f2b-j-w-iptables-new -j RETURN`", - "`ip6tables -w -I INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`{ ip6tables -w -C f2b-j-w-iptables-new -j RETURN >/dev/null 2>&1; } || " + "{ ip6tables -w -N f2b-j-w-iptables-new || true; ip6tables -w -A f2b-j-w-iptables-new -j RETURN; }`", + "`{ ip6tables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new >/dev/null 2>&1; } || " + "{ ip6tables -w -I INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new; }`", ), 'flush': ( "`iptables -w -F f2b-j-w-iptables-new`", "`ip6tables -w -F f2b-j-w-iptables-new`", ), 'stop': ( - "`iptables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`iptables -w -D INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`", "`iptables -w -F f2b-j-w-iptables-new`", "`iptables -w -X f2b-j-w-iptables-new`", - "`ip6tables -w -D INPUT -m state --state NEW -p tcp --dport http -j f2b-j-w-iptables-new`", + "`ip6tables -w -D INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`", "`ip6tables -w -F f2b-j-w-iptables-new`", "`ip6tables -w -X f2b-j-w-iptables-new`", ), 'ip4-check': ( - r"""`iptables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", + r"""`iptables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`""", ), 'ip6-check': ( - r"""`ip6tables -w -n -L INPUT | grep -q 'f2b-j-w-iptables-new[ \t]'`""", + r"""`ip6tables -w -C INPUT -m state --state NEW -p $proto --dport http -j f2b-j-w-iptables-new`""", ), 'ip4-ban': ( r"`iptables -w -I f2b-j-w-iptables-new 1 -s 192.0.2.1 -j REJECT --reject-with icmp-port-unreachable`", @@ -1754,22 +1792,26 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-iptables-xtre', 'iptables-xt_recent-echo[name=%(__name__)s, bantime="10m", chain="<known/chain>"]', { 'ip4': ('`iptables ', '/f2b-j-w-iptables-xtre`'), 'ip6': ('`ip6tables ', '/f2b-j-w-iptables-xtre6`'), 'ip4-start': ( - "`if [ `id -u` -eq 0 ];then iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + "`{ iptables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable >/dev/null 2>&1; } || { iptables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable; }`", ), 'ip6-start': ( - "`if [ `id -u` -eq 0 ];then ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", + "`{ ip6tables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable >/dev/null 2>&1; } || { ip6tables -w -I INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable; }`", ), 'stop': ( "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", - "`if [ `id -u` -eq 0 ];then iptables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;fi`", + "`if [ `id -u` -eq 0 ];then`", + "`iptables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable;`", + "`fi`", "`echo / > /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", - "`if [ `id -u` -eq 0 ];then ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;fi`", + "`if [ `id -u` -eq 0 ];then`", + "`ip6tables -w -D INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable;`", + "`fi`", ), 'ip4-check': ( - r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", + r"`{ iptables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre -j REJECT --reject-with icmp-port-unreachable; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre`", ), 'ip6-check': ( - r"`test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", + r"`{ ip6tables -w -C INPUT -m recent --update --seconds 3600 --name f2b-j-w-iptables-xtre6 -j REJECT --reject-with icmp6-port-unreachable; } && test -e /proc/net/xt_recent/f2b-j-w-iptables-xtre6`", ), 'ip4-ban': ( r"`echo +192.0.2.1 > /proc/net/xt_recent/f2b-j-w-iptables-xtre`", @@ -1937,11 +1979,11 @@ class ServerConfigReaderTests(LogCaptureTestCase): ('j-w-fwcmd-ipset', 'firewallcmd-ipset[name=%(__name__)s, port="http", protocol="tcp", chain="<known/chain>"]', { 'ip4': (' f2b-j-w-fwcmd-ipset ',), 'ip6': (' f2b-j-w-fwcmd-ipset6 ',), 'ip4-start': ( - "`ipset create f2b-j-w-fwcmd-ipset hash:ip timeout 0 `", + "`ipset -exist create f2b-j-w-fwcmd-ipset hash:ip timeout 0 `", "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( - "`ipset create f2b-j-w-fwcmd-ipset6 hash:ip timeout 0 family inet6`", + "`ipset -exist create f2b-j-w-fwcmd-ipset6 hash:ip timeout 0 family inet6`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m multiport --dports http -m set --match-set f2b-j-w-fwcmd-ipset6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( @@ -1957,27 +1999,27 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ipset destroy f2b-j-w-fwcmd-ipset6`", ), 'ip4-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset6 2001:db8::`", ), }), # firewallcmd-ipset (allports) -- ('j-w-fwcmd-ipset-ap', 'firewallcmd-ipset[name=%(__name__)s, actiontype=<allports>, protocol="tcp", chain="<known/chain>"]', { 'ip4': (' f2b-j-w-fwcmd-ipset-ap ',), 'ip6': (' f2b-j-w-fwcmd-ipset-ap6 ',), 'ip4-start': ( - "`ipset create f2b-j-w-fwcmd-ipset-ap hash:ip timeout 0 `", + "`ipset -exist create f2b-j-w-fwcmd-ipset-ap hash:ip timeout 0 `", "`firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap src -j REJECT --reject-with icmp-port-unreachable`", ), 'ip6-start': ( - "`ipset create f2b-j-w-fwcmd-ipset-ap6 hash:ip timeout 0 family inet6`", + "`ipset -exist create f2b-j-w-fwcmd-ipset-ap6 hash:ip timeout 0 family inet6`", "`firewall-cmd --direct --add-rule ipv6 filter INPUT_direct 0 -p tcp -m set --match-set f2b-j-w-fwcmd-ipset-ap6 src -j REJECT --reject-with icmp6-port-unreachable`", ), 'flush': ( @@ -1993,16 +2035,16 @@ class ServerConfigReaderTests(LogCaptureTestCase): "`ipset destroy f2b-j-w-fwcmd-ipset-ap6`", ), 'ip4-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset-ap 192.0.2.1 timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset-ap 192.0.2.1 timeout 0`", ), 'ip4-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset-ap 192.0.2.1 -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset-ap 192.0.2.1`", ), 'ip6-ban': ( - r"`ipset add f2b-j-w-fwcmd-ipset-ap6 2001:db8:: timeout 0 -exist`", + r"`ipset -exist add f2b-j-w-fwcmd-ipset-ap6 2001:db8:: timeout 0`", ), 'ip6-unban': ( - r"`ipset del f2b-j-w-fwcmd-ipset-ap6 2001:db8:: -exist`", + r"`ipset -exist del f2b-j-w-fwcmd-ipset-ap6 2001:db8::`", ), }), # firewallcmd-rich-rules -- @@ -2077,27 +2119,40 @@ class ServerConfigReaderTests(LogCaptureTestCase): # test ban ip4 : self.pruneLog('# === ban-ipv4 ===') action.ban(aInfos['ipv4']) - if tests.get('ip4-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip4-start'], all=True) + if tests.get('ip4-start'): self.assertLogged(*tests.get('*-start', tests.get('*-start-stop-check', ()))+tests['ip4-start'], all=True) if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True) - self.assertLogged(*tests.get('ip4-check',())+tests['ip4-ban'], all=True) + self.assertLogged(*tests['ip4-ban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test unban ip4 : self.pruneLog('# === unban ipv4 ===') action.unban(aInfos['ipv4']) - self.assertLogged(*tests.get('ip4-check',())+tests['ip4-unban'], all=True) + self.assertLogged(*tests['ip4-unban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test ban ip6 : self.pruneLog('# === ban ipv6 ===') action.ban(aInfos['ipv6']) - if tests.get('ip6-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip6-start'], all=True) + if tests.get('ip6-start'): self.assertLogged(*tests.get('*-start', tests.get('*-start-stop-check', ()))+tests['ip6-start'], all=True) if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True) - self.assertLogged(*tests.get('ip6-check',())+tests['ip6-ban'], all=True) + self.assertLogged(*tests['ip6-ban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) # test unban ip6 : self.pruneLog('# === unban ipv6 ===') action.unban(aInfos['ipv6']) - self.assertLogged(*tests.get('ip6-check',())+tests['ip6-unban'], all=True) + self.assertLogged(*tests['ip6-unban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) + # test invariant check (normally on demand in error case only): + if tests.get('ip4-check'): + self.pruneLog('# === check ipv4 ===') + action._invariantCheck(aInfos['ipv4']['family']) + self.assertLogged(*tests.get('*-check', tests.get('*-start-stop-check', ()))+tests['ip4-check'], all=True) + if tests.get('ip6-check') and tests['ip6-check'] != tests['ip4-check']: + self.assertNotLogged(*tests['ip6-check'], all=True) + if tests.get('ip6-check'): + self.pruneLog('# === check ipv6 ===') + action._invariantCheck(aInfos['ipv6']['family']) + self.assertLogged(*tests.get('*-check', tests.get('*-start-stop-check', ()))+tests['ip6-check'], all=True) + if tests.get('ip4-check') and tests['ip4-check'] != tests['ip6-check']: + self.assertNotLogged(*tests['ip4-check'], all=True) # test flush for actions should supported this: if tests.get('flush'): self.pruneLog('# === flush ===') @@ -2106,7 +2161,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): # test stop : self.pruneLog('# === stop ===') action.stop() - if tests.get('stop'): self.assertLogged(*tests['stop'], all=True) + if tests.get('stop'): self.assertLogged(*tests.get('*-start-stop-check', ())+tests['stop'], all=True) def _executeMailCmd(self, realCmd, timeout=60): # replace pipe to mail with pipe to cat: diff --git a/fail2ban/tests/tickettestcase.py b/fail2ban/tests/tickettestcase.py index d7d5f19a..771d2b50 100644 --- a/fail2ban/tests/tickettestcase.py +++ b/fail2ban/tests/tickettestcase.py @@ -39,6 +39,7 @@ class TicketTests(unittest.TestCase): # Ticket t = Ticket('193.168.0.128', tm, matches) + self.assertEqual(t.getID(), '193.168.0.128') self.assertEqual(t.getIP(), '193.168.0.128') self.assertEqual(t.getTime(), tm) self.assertEqual(t.getMatches(), matches2) @@ -65,6 +66,7 @@ class TicketTests(unittest.TestCase): matches = ['first', 'second'] ft = FailTicket('193.168.0.128', tm, matches) ft.setBanTime(60*60) + self.assertEqual(ft.getID(), '193.168.0.128') self.assertEqual(ft.getIP(), '193.168.0.128') self.assertEqual(ft.getTime(), tm) self.assertEqual(ft.getMatches(), matches2) @@ -116,6 +118,17 @@ class TicketTests(unittest.TestCase): self.assertEqual(ft2.getTime(), ft.getTime()) self.assertEqual(ft2.getBanTime(), ft.getBanTime()) + def testDiffIDAndIPTicket(self): + tm = MyTime.time() + # different ID (string) and IP: + t = Ticket('123-456-678', tm, data={'ip':'192.0.2.1'}) + self.assertEqual(t.getID(), '123-456-678') + self.assertEqual(t.getIP(), '192.0.2.1') + # different ID (tuple) and IP: + t = Ticket(('192.0.2.1', '5000'), tm, data={'ip':'192.0.2.1'}) + self.assertEqual(t.getID(), ('192.0.2.1', '5000')) + self.assertEqual(t.getIP(), '192.0.2.1') + def testTicketFlags(self): flags = ('restored', 'banned') ticket = Ticket('test', 0) diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index e674ee9b..8bcc1431 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -255,23 +255,6 @@ def with_alt_time(f): return wrapper -# backwards compatibility to python 2.6: -if not hasattr(unittest, 'SkipTest'): # pragma: no cover - class SkipTest(Exception): - pass - unittest.SkipTest = SkipTest - _org_AddError = unittest._TextTestResult.addError - def addError(self, test, err): - if err[0] is SkipTest: - if self.showAll: - self.stream.writeln(str(err[1])) - elif self.dots: - self.stream.write('s') - self.stream.flush() - return - _org_AddError(self, test, err) - unittest._TextTestResult.addError = addError - def initTests(opts): ## if running from installer (setup.py): if not opts: diff --git a/fail2ban/version.py b/fail2ban/version.py index ca799fcd..078e47ef 100644 --- a/fail2ban/version.py +++ b/fail2ban/version.py @@ -24,7 +24,7 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black" __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black" __license__ = "GPL-v2+" -version = "0.11.2" +version = "1.0.1.dev1" def normVersion(): """ Returns fail2ban version in normalized machine-readable format""" diff --git a/man/fail2ban-client.1 b/man/fail2ban-client.1 index 1cea4c7f..80e7fa48 100644 --- a/man/fail2ban-client.1 +++ b/man/fail2ban-client.1 @@ -1,12 +1,12 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-CLIENT "1" "November 2020" "fail2ban-client v0.11.2" "User Commands" +.TH FAIL2BAN-CLIENT "1" "February 2020" "fail2ban-client v1.0.1.dev1" "User Commands" .SH NAME fail2ban-client \- configure and control the server .SH SYNOPSIS .B fail2ban-client [\fI\,OPTIONS\/\fR] \fI\,<COMMAND>\/\fR .SH DESCRIPTION -Fail2Ban v0.11.2 reads log file that contains password failure report +Fail2Ban v1.0.1.dev1 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP @@ -165,7 +165,7 @@ logtarget is SYSLOG gets syslog socket path .TP \fBflushlogs\fR -flushes the logtarget if a file +flushes the logtarget of a file and reopens it. For log rotation. .IP DATABASE @@ -415,7 +415,7 @@ gets the time a host is banned for <JAIL> .TP \fBget <JAIL> datepattern\fR -gets the patern used to match +gets the pattern used to match date/times for <JAIL> .TP \fBget <JAIL> usedns\fR diff --git a/man/fail2ban-python.1 b/man/fail2ban-python.1 index 00b99403..5237f0fc 100644 --- a/man/fail2ban-python.1 +++ b/man/fail2ban-python.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-PYTHON "1" "November 2020" "fail2ban-python 0.11.2" "User Commands" +.TH FAIL2BAN-PYTHON "1" "January 2020" "fail2ban-python 1.0.1.1" "User Commands" .SH NAME fail2ban-python \- a helper for Fail2Ban to assure that the same Python is used .SH DESCRIPTION diff --git a/man/fail2ban-regex.1 b/man/fail2ban-regex.1 index 3bb0ca31..97f94d47 100644 --- a/man/fail2ban-regex.1 +++ b/man/fail2ban-regex.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-REGEX "1" "November 2020" "fail2ban-regex 0.11.2" "User Commands" +.TH FAIL2BAN-REGEX "1" "January 2020" "fail2ban-regex 1.0.1.dev1" "User Commands" .SH NAME fail2ban-regex \- test Fail2ban "failregex" option .SH SYNOPSIS diff --git a/man/fail2ban-server.1 b/man/fail2ban-server.1 index c18011cc..08745d72 100644 --- a/man/fail2ban-server.1 +++ b/man/fail2ban-server.1 @@ -1,12 +1,12 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-SERVER "1" "November 2020" "fail2ban-server v0.11.2" "User Commands" +.TH FAIL2BAN-SERVER "1" "February 2020" "fail2ban-server v1.0.1.dev1" "User Commands" .SH NAME fail2ban-server \- start the server .SH SYNOPSIS .B fail2ban-server [\fI\,OPTIONS\/\fR] .SH DESCRIPTION -Fail2Ban v0.11.2 reads log file that contains password failure report +Fail2Ban v1.0.1.dev1 reads log file that contains password failure report and bans the corresponding IP addresses using firewall rules. .SH OPTIONS .TP diff --git a/man/fail2ban-testcases.1 b/man/fail2ban-testcases.1 index dbdb190b..7a6fa4e4 100644 --- a/man/fail2ban-testcases.1 +++ b/man/fail2ban-testcases.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. -.TH FAIL2BAN-TESTCASES "1" "November 2020" "fail2ban-testcases 0.11.2" "User Commands" +.TH FAIL2BAN-TESTCASES "1" "January 2020" "fail2ban-testcases 1.0.1.dev1" "User Commands" .SH NAME fail2ban-testcases \- run Fail2Ban unit-tests .SH SYNOPSIS diff --git a/man/jail.conf.5 b/man/jail.conf.5 index 5f29161d..052fce80 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -271,7 +271,7 @@ effective ban duration (in seconds or time abbreviation format). time interval (in seconds or time abbreviation format) before the current time where failures will count towards a ban. .TP .B maxretry -number of failures that have to occur in the last \fBfindtime\fR seconds to ban then IP. +number of failures that have to occur in the last \fBfindtime\fR seconds to ban the IP. .TP .B backend backend to be used to detect changes in the logpath. @@ -487,7 +487,7 @@ is the regex (\fBreg\fRular \fBex\fRpression) that will match failed attempts. T .IP \fI<CIDR>\fR - helper regex to match CIDR (simple integer form of net-mask). .IP -\fI<SUBNET>\fR - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional). +\fI<SUBNET>\fR - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional). .IP \fI<F-ID>...</F-ID>\fR - free regex capturing group targeting identifier used for ban (instead of IP address or hostname). .IP @@ -540,7 +540,7 @@ There are several prefixes and words with special meaning that could be specifie .IP \fI{UNB}\fR - prefix to disable automatic word boundaries in regex. .IP -\fI{NONE}\fR - value would allow to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp). +\fI{NONE}\fR - value would allow one to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp). .RE .TP \fBjournalmatch\fR |