Merge branch 'master' into 0.9

Conflicts:
	ChangeLog
	bin/fail2ban-regex
	bin/fail2ban-testcases
	config/jail.conf
	fail2ban/server/failregex.py
	fail2ban/server/filter.py
	fail2ban/tests/files/logs/lighttpd
	fail2ban/tests/files/logs/mysqld.log
	fail2ban/tests/files/logs/wu-ftpd
	fail2ban/tests/filtertestcase.py
	fail2ban/tests/utils.py
	testcases/files/logs/lighttpd
	testcases/files/logs/lighttpd-auth
	testcases/files/logs/mysqld-auth
	testcases/files/logs/mysqld.log
	testcases/files/logs/wu-ftpd
	testcases/files/logs/wuftpd

1 files changed, 121 insertions, 4 deletions

diff --git a/DEVELOP b/DEVELOP
index 35d3b579..996a9552 100644
--- a/DEVELOP
+++ b/DEVELOP
@@ -39,10 +39,123 @@ Filters
 
 * Include sample logs with 1.2.3.4 used for IP addresses and 
   example.com/example.org used for DNS names
-* Ensure ./fail2ban-regex testcases/files/logs/{samplelog} config/filter.d/{filter}.conf
-  has matches for EVERY regex
-* Ensure regexs end with a $ and are restrictive as possible. E.g. not .* if
- [0-9]+ is sufficient
+* Ensure sample log is provided in testcases/files/logs/ with same name as the
+  filter. Each log line should include match meta data for time & IP above
+  every line (see other sample log files for examples)
+* Ensure regexs start with a ^ and are restrictive as possible. E.g. not .* if
+  \d+ is sufficient
+* Use the functionality of regexs http://docs.python.org/2/library/re.html
+* Take a look at the source code of the application. You may see optional or
+  extra log messages, or parts there of, that need to form part of your regex.
+
+If you only have a basic knowledge of regular repressions read
+http://docs.python.org/2/library/re.html first.
+
+Filter Security
+---------------
+
+Poor filter regular expressions are suseptable to DoS attacks.
+
+When a remote user has the ability to introduce text that will match the 
+filter regex, such that the inserted text matches the <HOST> part, they have the
+ability to deny any host they choose.
+
+So the <HOST> part must be anchored on text generated by the application, and not
+the user, to a sufficient extent that the user cannot insert the entire text.
+
+Filters are matched against the log line with their date removed.
+
+Ideally filter regex should anchor to the beginning and end of the log line
+however as more applications log at the beginning than the end, achoring the
+beginning is more important. If the log file used by the application is shared
+with other applications, like system logs, ensure the other application that
+use that log file do not log user generated text at the beginning of the line,
+or, if they do, ensure the regexs of the filter are sufficient to mitigate the
+risk of insertion.
+
+When creating a regex that extends back to the begining remember the date part
+has been removed within fail2ban so theres no need to match that. If the format
+is like '<date...> error 1.2.3.4 is evil' then you will need to match the < at
+the start so here the regex would start like '^<> <HOST> is evil$'.
+
+Some applications log spaces at the end. If you're not sure add \s*$ as the
+end part of the regex.
+
+Examples of poor filters
+------------------------
+
+1. Too restrictive
+
+We find a log message:
+
+    Apr-07-13 07:08:36 Invalid command fial2ban from 1.2.3.4
+
+We make a failregex
+
+    ^Invalid command \S+ from <HOST>
+
+Now think evil. The user does the command 'blah from 1.2.3.44'
+
+The program diliently logs:
+
+    Apr-07-13 07:08:36 Invalid command blah from 1.2.3.44 from 1.2.3.4
+
+And fail2ban matches 1.2.3.44 as the IP that it ban. A DoS attack was successful.
+
+The fix here is that the command can be anything so .* is approprate.
+
+    ^Invalid command .* from <HOST>
+
+Here the .* will match until the end of the string. Then realise it has more to
+match, i.e. "from <HOST>" and go back until it find this. Then it will ban
+1.2.3.4 correctly. Since the <HOST> is always at the end, end the regex with a $.
+
+    ^Invalid command .* from <HOST>$
+
+Note if we'd just had the expression:
+
+    ^Invalid command \S+ from <HOST>$
+
+Then provided the user put a space in their command they would have never been
+banned.
+
+2. Filter regex can match other user injected data
+
+From the apache vulnerability CVE-2013-2178
+( original ref: https://vndh.net/note:fail2ban-089-denial-service ).
+
+An example bad regex for apache:
+
+    failregex = [[]client <HOST>[]] user .* not found
+
+Since the user can do a get request on:
+
+    GET /[client%20192.168.0.1]%20user%20root%20not%20found HTTP/1.0
+Host: remote.site
+
+Now the log line will be:
+
+    [Sat Jun 01 02:17:42 2013] [error] [client 192.168.33.1] File does not exist: /srv/http/site/[client 192.168.0.1] user root not found
+
+As this log line doesn't match other expressions hence it matches the above
+regex and blocks 192.168.33.1 as a denial of service from the HTTP requester.
+
+3. Applicaiton generates two identical log messages with different meanings
+
+If the application generates the following two messages under different
+circmstances:
+
+    client <IP>: authentication failed
+    client <USER>: authentication failed
+
+
+Then it's obvious that a regex of "^client <HOST>: authentication
+failed$" will still cause problems if the user can trigger the second
+log message with a <USER> of 123.1.1.1.
+
+Here there's nothing to do except request/change the application so it logs
+messages differently.
+
 
 Code Testing
 ============
@@ -136,6 +249,10 @@ Use the following tags in your commit messages:
 
 Multiple tags could be joined with +, e.g. "BF+TST:".
 
+Use the text "closes #333"/"resolves #333 "/"fixes #333" where 333 represents
+an issue that is closed. Other text and details in link below.
+See: https://help.github.com/articles/closing-issues-via-commit-messages
+
 Adding Actions
 --------------