BreakIterator and RuleBasedBreakiterator added

This commit adds wrappers for the classes BreakIterator and
RuleBasedbreakIterator. The C++ ICU classes are described here:
<http://icu-project.org/apiref/icu4c/classBreakIterator.html>
<http://icu-project.org/apiref/icu4c/classRuleBasedBreakIterator.html>

Additionally, a tutorial is available at:
<http://userguide.icu-project.org/boundaryanalysis>

This implementation wraps UTF-8 text in a UText. The text is
iterated without any copying or conversion to UTF-16. There is
also no validation that the input is actually UTF-8; where there
are malformed sequences, the UText will simply U+FFFD.

The class BreakIterator cannot be instantiated directly (has a
private constructor). It provides the interface exposed by the ICU
abstract class with the same name. The PHP class is not abstract
because we may use it to wrap native subclasses of BreakIterator
that we don't know how to wrap. This class includes methods to
move the iterator position to the beginning (first()), to the
end (last()), forward (next()), backwards (previous()), to the
boundary preceding a certain position (preceding()) and following
a certain position (following()) and to obtain the current position
(current()). next() can also be used to advance or recede an
arbitrary number of positions.

BreakIterator also exposes other native methods:
getAvailableLocales(), getLocale() and factory methods to build
several predefined types of BreakIterators: createWordInstance()
for word boundaries, createCharacterInstance() for locale
dependent notions of "characters", createSentenceInstance() for
sentences, createLineInstance() and createTitleInstance() -- for
title casing breaks. These factories currently return
RuleBasedbreakIterators where the names of the rule sets are found
in the ICU data, observing the passed locale (although the locale
is taken into considering there are very few exceptions to the
root rules).

The clone and compare_object PHP object handlers are also
implemented, though the comparison does not yield meaningful results
when used with >, <, >= and <=.

Note that BreakIterator is an iterator only in the sense of the
first 'Iterator' in 'IteratorIterator', i.e., it does not
implement the Iterator interface. The reason is that there is
no sensible implementation for Iterator::key(). Using it for
an ordinal of the current boundary is not feasible because
we are allowed to move to any boundary at any time. It we were
to determine the current ordinal when last() is called we'd
have to traverse the whole input text to find out how many
breaks there were before. Therefore, BreakIterator implements
only Traversable. It can be wrapped in an IteratorIterator,
but the usual warnings apply.

Finally, I added a convenience method to BreakIterator:
getPartsIterator(). This provides an IntlIterator, backed
by the BreakIterator PHP object (i.e. moving the pointer or
changing the text in BreakIterator affects the iterator
and also moving the iterator affects the backing BreakIterator),
which allows traversing the text between each boundary.
This iterator uses the original text to retrieve the text
between two positions, not the code points returned by the
wrapping UText. Therefore, if the text includes invalid code
unit sequences, these invalid sequences will be in the output
of this iterator, not U+FFFD code points.

The class RuleBasedIterator exposes a constructor that allows
building an iterator from arbitrary compiled or non-compiled
rules. The form of these rules in described in the tutorial linked
above. The rest of the methods allow retrieving the rules --
getRules() and getCompiledRules() --, a hash code of the rule set
(hashCode()) and the rules statuses (getRuleStatus() and
getRuleStatusVec()).

Because the RuleBasedBreakIterator constructor may return parse
errors, I reuse the UParseError to text function that was in the
transliterator files. Therefore, I move that function to
intl_error.c.

common_enum.cpp was also changed, mainly to expose previously
static functions. This avoided code duplication when implementing
the BreakIterator iterator and the IntlIterator returned by
BreakIterator::getPartsIterator().

1 files changed, 442 insertions, 0 deletions

diff --git a/ext/intl/breakiterator/breakiterator_methods.cpp b/ext/intl/breakiterator/breakiterator_methods.cpp
new file mode 100644
index 0000000000..4aca6ef23f
--- /dev/null
+++ b/ext/intl/breakiterator/breakiterator_methods.cpp
@@ -0,0 +1,442 @@
+/*
+   +----------------------------------------------------------------------+
+   | PHP Version 5                                                        |
+   +----------------------------------------------------------------------+
+   | This source file is subject to version 3.01 of the PHP license,      |
+   | that is bundled with this package in the file LICENSE, and is        |
+   | available through the world-wide-web at the following url:           |
+   | http://www.php.net/license/3_01.txt                                  |
+   | If you did not receive a copy of the PHP license and are unable to   |
+   | obtain it through the world-wide-web, please send a note to          |
+   | license@php.net so we can mail you a copy immediately.               |
+   +----------------------------------------------------------------------+
+   | Authors: Gustavo Lopes <cataphract@php.net>                          |
+   +----------------------------------------------------------------------+
+*/
+
+#ifdef HAVE_CONFIG_H
+#include "config.h"
+#endif
+
+#include <unicode/brkiter.h>
+
+#include "breakiterator_iterators.h"
+
+extern "C" {
+#define USE_BREAKITERATOR_POINTER 1
+#include "breakiterator_class.h"
+#include "../locale/locale.h"
+#include <zend_exceptions.h>
+}
+
+U_CFUNC PHP_METHOD(BreakIterator, __construct)
+{
+	zend_throw_exception( NULL,
+		"An object of this type cannot be created with the new operator",
+		0 TSRMLS_CC );
+}
+
+static void _breakiter_factory(const char *func_name,
+							   BreakIterator *(*func)(const Locale&, UErrorCode&),
+							   INTERNAL_FUNCTION_PARAMETERS)
+{
+	BreakIterator	*biter;
+	const char		*locale_str = NULL;
+	int				dummy;
+	char			*msg;
+	UErrorCode		status = UErrorCode();
+	intl_error_reset(NULL TSRMLS_CC);
+	
+	if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "s!",
+			&locale_str, &dummy) == FAILURE) {
+		spprintf(&msg, NULL, "%s: bad arguments", func_name);
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR, msg, 1 TSRMLS_CC);
+		efree(msg);
+		RETURN_NULL();
+	}
+	
+	if (locale_str == NULL) {
+		locale_str = intl_locale_get_default(TSRMLS_C);
+	}
+	
+	biter = func(Locale::createFromName(locale_str), status);
+	intl_error_set_code(NULL, status TSRMLS_CC);
+	if (U_FAILURE(status)) {
+		spprintf(&msg, NULL, "%s: error creating BreakIterator",
+				func_name);
+		intl_error_set_custom_msg(NULL, msg, 1 TSRMLS_CC);
+		efree(msg);
+		RETURN_NULL();
+	}
+	
+	breakiterator_object_create(return_value, biter TSRMLS_CC);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_create_word_instance)
+{
+	_breakiter_factory("breakiter_create_word_instance",
+			&BreakIterator::createWordInstance,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_create_line_instance)
+{
+	_breakiter_factory("breakiter_create_line_instance",
+			&BreakIterator::createLineInstance,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_create_character_instance)
+{
+	_breakiter_factory("breakiter_create_character_instance",
+			&BreakIterator::createCharacterInstance,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_create_sentence_instance)
+{
+	_breakiter_factory("breakiter_create_sentence_instance",
+			&BreakIterator::createSentenceInstance,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_create_title_instance)
+{
+	_breakiter_factory("breakiter_create_title_instance",
+			&BreakIterator::createTitleInstance,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_get_available_locales)
+{
+	intl_error_reset(NULL TSRMLS_CC);
+	
+	if (zend_parse_parameters_none() == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+				"breakiter_get_available_locales: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+	
+	const	Locale *locales;
+	int32_t	count;
+	
+	locales = BreakIterator::getAvailableLocales(count);
+	array_init_size(return_value, (uint)count);
+	for (int i = 0; i < count; i++) {
+		Locale locale = locales[i];
+		add_next_index_string(return_value, locale.getName(), 1);
+	}
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_get_text)
+{
+	BREAKITER_METHOD_INIT_VARS;
+
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "O",
+			&object, BreakIterator_ce_ptr) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+			"breakiter_get_text: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+	
+	BREAKITER_METHOD_FETCH_OBJECT;
+	
+	if (bio->text == NULL) {
+		RETURN_NULL();
+	} else {
+		RETURN_ZVAL(bio->text, 1, 0);
+	}
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_set_text)
+{
+	char	*text;
+	int		text_len;
+	UText	*ut = NULL;
+	zval	**textzv;
+	BREAKITER_METHOD_INIT_VARS;
+	
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "Os",
+			&object, BreakIterator_ce_ptr, &text, &text_len) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+			"breakiter_set_text: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+	
+	int res = zend_get_parameters_ex(1, &textzv);
+	assert(res == SUCCESS);
+	
+	BREAKITER_METHOD_FETCH_OBJECT;
+	
+	/* assert it's safe to use text and text_len because zpp changes the
+	 * arguments in the stack */
+	assert(text == Z_STRVAL_PP(textzv));
+	
+	ut = utext_openUTF8(ut, text, text_len, BREAKITER_ERROR_CODE_P(bio));
+	INTL_CTOR_CHECK_STATUS(bio, "breakiter_set_text: error opening UText");
+	
+	bio->biter->setText(ut, BREAKITER_ERROR_CODE(bio));
+	utext_close(ut); /* ICU shallow clones the UText */
+	INTL_CTOR_CHECK_STATUS(bio, "breakiter_set_text: error calling "
+		"BreakIterator::setText()");
+	
+	/* When ICU clones the UText, it does not copy the buffer, so we have to
+	 * keep the string buffer around by holding a reference to its zval. This
+	 * also allows a faste implementation of getText() */
+	if (bio->text != NULL) {
+		zval_ptr_dtor(&bio->text);
+	}
+	bio->text = *textzv;
+	zval_add_ref(&bio->text);
+	
+	RETURN_TRUE;
+}
+
+static void _breakiter_no_args_ret_int32(
+		const char *func_name,
+		int32_t (BreakIterator::*func)(),
+		INTERNAL_FUNCTION_PARAMETERS)
+{
+	char	*msg;
+	BREAKITER_METHOD_INIT_VARS;
+	
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "O",
+			&object, BreakIterator_ce_ptr) == FAILURE) {
+		spprintf(&msg, NULL, "%s: bad arguments", func_name);
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR, msg, 1 TSRMLS_CC);
+		efree(msg);
+		RETURN_FALSE;
+	}
+	
+	BREAKITER_METHOD_FETCH_OBJECT;
+	
+	int32_t res = (bio->biter->*func)();
+	
+	RETURN_LONG((long)res);
+}
+
+static void _breakiter_int32_ret_int32(
+		const char *func_name,
+		int32_t (BreakIterator::*func)(int32_t),
+		INTERNAL_FUNCTION_PARAMETERS)
+{
+	char	*msg;
+	long	arg;
+	BREAKITER_METHOD_INIT_VARS;
+	
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "Ol",
+			&object, BreakIterator_ce_ptr, &arg) == FAILURE) {
+		spprintf(&msg, NULL, "%s: bad arguments", func_name);
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR, msg, 1 TSRMLS_CC);
+		efree(msg);
+		RETURN_FALSE;
+	}
+	
+	BREAKITER_METHOD_FETCH_OBJECT;
+	
+	if (arg < INT32_MIN || arg > INT32_MAX) {
+		spprintf(&msg, NULL, "%s: offset argument is outside bounds of "
+				"a 32-bit wide integer", func_name);
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR, msg, 1 TSRMLS_CC);
+		efree(msg);
+		RETURN_FALSE;
+	}
+	
+	int32_t res = (bio->biter->*func)((int32_t)arg);
+	
+	RETURN_LONG((long)res);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_first)
+{
+	_breakiter_no_args_ret_int32("breakiter_first",
+			&BreakIterator::first,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_last)
+{
+	_breakiter_no_args_ret_int32("breakiter_last",
+			&BreakIterator::last,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_previous)
+{
+	_breakiter_no_args_ret_int32("breakiter_previous",
+			&BreakIterator::previous,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_next)
+{
+	bool no_arg_version = false;
+	
+	if (ZEND_NUM_ARGS() == 0) {
+		no_arg_version = true;
+	} else if (ZEND_NUM_ARGS() == 1) {
+		zval **arg;
+		int res = zend_get_parameters_ex(1, &arg);
+		assert(res == SUCCESS);
+		if (Z_TYPE_PP(arg) == IS_NULL) {
+			no_arg_version = true;
+			ht = 0; /* pretend we don't have any argument */
+		} else {
+			no_arg_version = false;
+		}
+	}
+	
+	if (no_arg_version) {
+		_breakiter_no_args_ret_int32("breakiter_next",
+				&BreakIterator::next,
+				INTERNAL_FUNCTION_PARAM_PASSTHRU);
+	} else {
+		_breakiter_int32_ret_int32("breakiter_next",
+				&BreakIterator::next,
+				INTERNAL_FUNCTION_PARAM_PASSTHRU);
+	}
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_current)
+{
+	BREAKITER_METHOD_INIT_VARS;
+	
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "O",
+			&object, BreakIterator_ce_ptr) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+				"breakiter_current: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+	
+	BREAKITER_METHOD_FETCH_OBJECT;
+	
+	int32_t res = bio->biter->current();
+	
+	RETURN_LONG((long)res);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_following)
+{
+	_breakiter_int32_ret_int32("breakiter_following",
+			&BreakIterator::following,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_preceding)
+{
+	_breakiter_int32_ret_int32("breakiter_preceding",
+			&BreakIterator::preceding,
+			INTERNAL_FUNCTION_PARAM_PASSTHRU);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_is_boundary)
+{
+	long offset;
+	BREAKITER_METHOD_INIT_VARS;
+	
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "Ol",
+			&object, BreakIterator_ce_ptr, &offset) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+				"breakiter_is_boundary: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+	
+	if (offset < INT32_MIN || offset > INT32_MAX) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+				"breakiter_is_boundary: offset argument is outside bounds of "
+				"a 32-bit wide integer", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+	
+	BREAKITER_METHOD_FETCH_OBJECT;
+	
+	UBool res = bio->biter->isBoundary((int32_t)offset);
+	
+	RETURN_BOOL((long)res);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_get_locale)
+{
+	long	locale_type;
+	BREAKITER_METHOD_INIT_VARS;
+
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(),
+			"Ol", &object, BreakIterator_ce_ptr, &locale_type) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+			"breakiter_get_locale: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+
+	if (locale_type != ULOC_ACTUAL_LOCALE && locale_type != ULOC_VALID_LOCALE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+			"breakiter_get_locale: invalid locale type", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+
+	BREAKITER_METHOD_FETCH_OBJECT;
+
+	Locale locale = bio->biter->getLocale((ULocDataLocaleType)locale_type,
+		BREAKITER_ERROR_CODE(bio));
+	INTL_METHOD_CHECK_STATUS(bio,
+		"breakiter_get_locale: Call to ICU method has failed");
+
+	RETURN_STRING(locale.getName(), 1);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_get_parts_iterator)
+{
+	BREAKITER_METHOD_INIT_VARS;
+	
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "O",
+			&object, BreakIterator_ce_ptr) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+				"breakiter_get_parts_iterator: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+
+	BREAKITER_METHOD_FETCH_OBJECT;
+
+	IntlIterator_from_BreakIterator_parts(object, return_value TSRMLS_CC);
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_get_error_code)
+{
+	BREAKITER_METHOD_INIT_VARS;
+
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "O",
+			&object, BreakIterator_ce_ptr) == FAILURE) {
+		intl_error_set(NULL, U_ILLEGAL_ARGUMENT_ERROR,
+			"breakiter_get_error_code: bad arguments", 0 TSRMLS_CC);
+		RETURN_FALSE;
+	}
+
+	/* Fetch the object (without resetting its last error code ). */
+	bio = (BreakIterator_object*)zend_object_store_get_object(object TSRMLS_CC);
+	if (bio == NULL)
+		RETURN_FALSE;
+
+	RETURN_LONG((long)BREAKITER_ERROR_CODE(bio));
+}
+
+U_CFUNC PHP_FUNCTION(breakiter_get_error_message)
+{
+	const char* message = NULL;
+	BREAKITER_METHOD_INIT_VARS;
+
+	if (zend_parse_method_parameters(ZEND_NUM_ARGS() TSRMLS_CC, getThis(), "O",
+			&object, BreakIterator_ce_ptr) == FAILURE) {
+		intl_error_set( NULL, U_ILLEGAL_ARGUMENT_ERROR,
+			"breakiter_get_error_message: bad arguments", 0 TSRMLS_CC );
+		RETURN_FALSE;
+	}
+
+
+	/* Fetch the object (without resetting its last error code ). */
+	bio = (BreakIterator_object*)zend_object_store_get_object(object TSRMLS_CC);
+	if (bio == NULL)
+		RETURN_FALSE;
+
+	/* Return last error message. */
+	message = intl_error_get_message(BREAKITER_ERROR_P(bio) TSRMLS_CC);
+	RETURN_STRING(message, 0);
+}