diff options
40 files changed, 950 insertions, 1600 deletions
@@ -46,6 +46,9 @@ one of the core developers (members of <http://launchpad.net/~zeitgeist>) Release Tasks ************* +0. Ensure the translations are up-to-date. You may find daily translation + exports provided by Launchpad at lp:~rainct/zeitgeist/translations. + 1. Bump version number to $VERSION in configure.ac and VERSION files, and the D-Bus API version in _zeitgeist/engine/remote.py, if there have been changes to it. Also make sure to bump the 'version' and @@ -96,19 +99,20 @@ Release Tasks Bugs **** -The zeitgeist project is using the bugtracker on launchpad.net. To report -a bug please visit https://bugs.edge.launchpad.net/zeitgeist/+filebug -and describe the issues you are having. Please also add useful information -like the version of zeitgeist you are using and the python version. -Appart from classical bugreport describing an issue or failure, the -zeitgeist team is using the bugtracker to discuss new features and +The Zeitgeist project is using the bugtracker on launchpad.net. To report +a bug please visit https://bugs.launchpad.net/zeitgeist/+filebug and describe +the issues you are having. Please also add useful information +like the version of Zeitgeist you are using and the Python version. + +Apart from classical bug reports describing an issue or failure, the +Zeitgeist team is using the bugtracker to discuss new features and implementation details. -Bugtriaging -*********** +Bug triaging +************ -The zeitgeist project is using bug status as described below: - * New: no bugtriager had a look at this bugreport +The Zeitgeist project is using bug status as described below: + * New: no bug triager had a look at this bugreport * Incomplete: the zeitgeist team has asked an explicit question to get a better understanding of the bug and is waiting for an answer from the original reporter or anyobne having the same issue @@ -147,7 +151,7 @@ Milestones: we should target this bug to a milestone * 'High' and 'Critical' rated bugs will always be targeted to the next milestone. - + Related policies: * Every bug of status 'Triaged' or higher should have an importance other than 'Undecided'. diff --git a/Makefile.am b/Makefile.am index 43544961..669bb1c5 100644 --- a/Makefile.am +++ b/Makefile.am @@ -6,8 +6,7 @@ INTLTOOL = \ intltool-update.in bin_SCRIPTS = \ - zeitgeist-daemon \ - zeitgeist-datahub + zeitgeist-daemon EXTRA_DIST = \ $(bin_SCRIPTS) \ @@ -16,7 +15,6 @@ EXTRA_DIST = \ COPYRIGHT \ NEWS \ zeitgeist-daemon.py \ - zeitgeist-datahub.py \ zeitgeist-daemon.pc.in DISTCLEANFILES = \ @@ -25,8 +23,7 @@ DISTCLEANFILES = \ intltool-update CLEANFILES = \ - zeitgeist-daemon \ - zeitgeist-datahub + zeitgeist-daemon pkgconfigdir = $(libdir)/pkgconfig pkgconfig_DATA = zeitgeist-daemon.pc @@ -34,19 +31,12 @@ pkgconfig_DATA = zeitgeist-daemon.pc zeitgeist-daemon: zeitgeist-daemon.py sed \ -e "s!\/usr\/bin\/env python!$(PYTHON)!" \ - -e "s!zeitgeist-datahub\.py!zeitgeist-datahub!" \ < $< > $@ chmod +x zeitgeist-daemon zeitgeist-daemon: Makefile -zeitgeist-datahub: zeitgeist-datahub.py - sed \ - -e "s!\/usr\/bin\/env python!$(PYTHON)!" \ - < $< > $@ - chmod +x zeitgeist-datahub -zeitgeist-datahub: Makefile -all-local: zeitgeist-daemon zeitgeist-datahub +all-local: zeitgeist-daemon # Generate ChangeLog dist-hook: @@ -79,3 +69,14 @@ check-local: else \ echo "Cannot run the testsuite, tests are not shipped"; \ fi + +# do the right thing to start a daemon +# wrap commandline options +# * to run the daemon without any datahub, set ZG_NODATAHUB, +# * to use a certain log level, set ZG_LOGLEVEL=<log-level> +# example: +# make run ZG_LOGLEVEL=INFO ZG_NODATAHUB=1 +ZG_DAEMON_ARGS := $(if $(ZG_LOGLEVEL), --log-level $(ZG_LOGLEVEL),) +ZG_DAEMON_ARGS += $(if $(ZG_NODATAHUB), --no-datahub,) +run: all + ./zeitgeist-daemon $(ZG_DAEMON_ARGS) @@ -1,3 +1,39 @@ +2010-09-26: Zeitgeist 0.5.2 "Snowball Effect" +--------------------------------------------- + +Engine: + + - Extensions and the database connection are now shutdown gracefully when + the Zeitgeist daemon is asked to stop over D-Bus. + - Receiving a SIGHUP indicates a request for a clean shutdown; this + can be used to signal a system-wide restart request after an update. + - Fixed a bug in TableLookup because of which the internal cache wasn't + being used. + - Added a new option, activated by the ZEITGEIST_DEBUG_QUERY_PLANS, which + prints detailed information about each query being executed. + - Removed superfluous database commits; now they are called after all + InsertEvent actions have been completed, not once for every event. + - Speed up the initial query retrieving the last event ID by fixing it to + make use of the database's indexes. + - Catch any exceptions trying to notify a monitor of changes and redirect + them to the error log. + - Fixed LeastRecentActor sorting not working in combination with event + template filtering (LP: #641968). + - Renamed the LeastRecentActor sorting type to OldestActor and implemented + a new LeastRecentActor which does what it was supposed to do (LP: #646124). + +Datahub: + + - Added a workaround so that events concerning OpenOffice.org are logged + correctly (LP: #646724). + +Overall: + + - Fix an error in the ontology; it's "rdf:Property", not "rdfs:Property". + - Improved documentation; particularly, the ontology is now presented + in a separate section with a more readable format. + - Translation updates (Brazilian Portuguese, Catalan and Indonesian). + 2010-09-09: Zeitgeist 0.5.1 "Spongebob is not funny" ---------------------------------------------------- @@ -49,13 +85,16 @@ Engine: - Extensions found in the extensions directory are now automatically loaded. - Install a .pc file so out of tree extensions can figure out the install path. - Load extensions found in ~/.local/share/zeitgeist/extensions. - - Let the GtkRecentlyUsed data-source ignore any exceptions while trying to - parse .desktop files (LP: #523761). - Fix return value of the SetDataSourceEnabled D-Bus method. - Extensions: Hooks have been renamed and most of them now have pre and post variants (LP: #592599, #604747). - Add new ResultTypes for sorting by subject origin +Datahub: + + - Let the GtkRecentlyUsed data-source ignore any exceptions while trying to + parse .desktop files (LP: #523761). + Python API: - ZeitgeistDBusInterface.get_extension is no longer a classmethod (in fact, it never really was). @@ -101,9 +140,11 @@ Engine: can make use of the event ids Python API: + - TimeRange sprouted a few new useful methods. Ontology: + - Removed {Interpretation,Manifestation}.UNKNOWN. If you really really (like really!) can not come up with a good interpretation or manifestation don't log it - or insert an empty string instead. @@ -122,6 +163,7 @@ Ontology: ERROR_EVENT. Overall: + - Other fixes and code enhancements. - Manpage updates. - Translation updates. @@ -154,12 +196,6 @@ Engine: - Overhauled FindRelatedUris for better results and improved performance. - Changed FindEvents, FindEventIds and FindRelatedUris to not treat zeros in the given TimeRange specially; "(0, 0)" can no longer be used (LP: #490242). - - Fixed a crash in the GtkRecentlyUsed data-source parsing malfored .desktop - files (LP: #526357), and added support for more file mimetypes (LP: #510761). - - Fixed a crash in the GtkRecentlyUsed data-source trying to read broken - symlinks disguised as .desktop files (LP: #523761). - - Fixed a crash in the GtkRecentlyUsed data-source which happened when there - was no display friendly version of a URI (LP: #531793). - Renamed --no-passive-loggers option to --no-datahub. Output printed by zeitgeist-datahub is no longer visible in zeitgeist-daemon's output. - Added --log-level option to change the output verbosity. @@ -170,7 +206,17 @@ Engine: - Fixed event deletions not always getting committed (LP: #566184). - Ignore deletion requests for non-existant events. +Datahub: + + - Fixed a crash in the GtkRecentlyUsed data-source parsing malfored .desktop + files (LP: #526357), and added support for more file mimetypes (LP: #510761). + - Fixed a crash in the GtkRecentlyUsed data-source trying to read broken + symlinks disguised as .desktop files (LP: #523761). + - Fixed a crash in the GtkRecentlyUsed data-source which happened when there + was no display friendly version of a URI (LP: #531793). + Python API: + - Made the Interpretation and Manifestation classes iterable. - Added symbol lookup by URI, in the form of dictionary access. - Fixed the display name for Interpretation.SOURCECODE. @@ -179,6 +225,7 @@ Python API: access to D-Bus interfaces provided by engine extensions. Overall: + - More fixes and code enhancements. - Manpage updates. - Translation updates. @@ -196,9 +243,11 @@ Engine: - Fixed GetEvents not to raise an exception when called with an empty list. Python API: + - ZeitgeistClient.get_version() now returns a Python list. Overall: + - Some code refactoring, documentation changes and other little fixes. 2010-01-10: Zeitgeist 0.3.1 "Mystical Tremor" @@ -244,12 +293,14 @@ is a complete rework of Zeitgeist and applications written for 0.2 won't work without changes. Engine: + - Completely reworked the engine and DBus API. - Removed the Storm backend (obsoleted in 0.2.1) and the Querymancer backend. - Added support for extensions written in Python, with direct access to the database. Python API: + - Added a public Python client API split into two modules, zeitgeist.datamodel and zeitgeist.client. - Changed ontologies from XESAM to Nepomuk. @@ -1 +1 @@ -0.5.0 +0.5.2 diff --git a/_zeitgeist/Makefile.am b/_zeitgeist/Makefile.am index c82cb7f3..a18519c2 100644 --- a/_zeitgeist/Makefile.am +++ b/_zeitgeist/Makefile.am @@ -1,4 +1,4 @@ -SUBDIRS = engine loggers +SUBDIRS = engine appdir = $(datadir)/zeitgeist/_zeitgeist/ diff --git a/_zeitgeist/engine/__init__.py b/_zeitgeist/engine/__init__.py index 4d5aa320..bf299b67 100644 --- a/_zeitgeist/engine/__init__.py +++ b/_zeitgeist/engine/__init__.py @@ -31,7 +31,6 @@ __all__ = [ "constants" ] -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.engine") _engine = None diff --git a/_zeitgeist/engine/extension.py b/_zeitgeist/engine/extension.py index eeb797b3..85725709 100644 --- a/_zeitgeist/engine/extension.py +++ b/_zeitgeist/engine/extension.py @@ -22,7 +22,6 @@ import os import logging import weakref # avoid circular references as they confuse garbage collection -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.extension") import zeitgeist diff --git a/_zeitgeist/engine/extensions/blacklist.py b/_zeitgeist/engine/extensions/blacklist.py index 026d5a85..9293613b 100644 --- a/_zeitgeist/engine/extensions/blacklist.py +++ b/_zeitgeist/engine/extensions/blacklist.py @@ -28,7 +28,6 @@ from _zeitgeist.engine.datamodel import Event from _zeitgeist.engine.extension import Extension from _zeitgeist.engine import constants -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.blacklist") CONFIG_FILE = os.path.join(constants.DATA_PATH, "blacklist.pickle") diff --git a/_zeitgeist/engine/extensions/datasource_registry.py b/_zeitgeist/engine/extensions/datasource_registry.py index 04747aa0..93993a36 100644 --- a/_zeitgeist/engine/extensions/datasource_registry.py +++ b/_zeitgeist/engine/extensions/datasource_registry.py @@ -30,7 +30,6 @@ from _zeitgeist.engine.datamodel import Event, DataSource as OrigDataSource from _zeitgeist.engine.extension import Extension from _zeitgeist.engine import constants -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.datasource_registry") DATA_FILE = os.path.join(constants.DATA_PATH, "datasources.pickle") diff --git a/_zeitgeist/engine/main.py b/_zeitgeist/engine/main.py index 6b1c2422..064e1b8b 100644 --- a/_zeitgeist/engine/main.py +++ b/_zeitgeist/engine/main.py @@ -24,11 +24,7 @@ import sqlite3 import time import sys import os -import math -import gettext import logging -import operator -from itertools import islice from collections import defaultdict from zeitgeist.datamodel import Event as OrigEvent, StorageState, TimeRange, \ @@ -38,10 +34,7 @@ from _zeitgeist.engine.extension import ExtensionsCollection, get_extensions from _zeitgeist.engine import constants from _zeitgeist.engine.sql import get_default_cursor, unset_cursor, \ TableLookup, WhereClause - -WINDOW_SIZE = 7 -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.engine") class NegationNotSupported(ValueError): @@ -307,8 +300,11 @@ class ZeitgeistEngine: raise NotImplementedError where = WhereClause(WhereClause.AND) - where.add("timestamp >= ?", time_range[0]) - where.add("timestamp <= ?", time_range[1]) + min_time, max_time = time_range + if min_time != 0: + where.add("timestamp >= ?", min_time) + if max_time != sys.maxint: + where.add("timestamp <= ?", max_time) where.extend(self._build_sql_from_event_templates(templates)) @@ -324,7 +320,6 @@ class ZeitgeistEngine: Return modes: - 0: IDs. - 1: Events. - - 2: (Timestamp, SubjectUri) """ t = time.time() @@ -338,20 +333,17 @@ class ZeitgeistEngine: sql = "SELECT DISTINCT id FROM event_view" elif return_mode == 1: sql = "SELECT id FROM event_view" - elif return_mode == 2: - sql = "SELECT subj_uri, timestamp FROM event_view" else: raise NotImplementedError, "Unsupported return_mode." - if order == ResultType.LeastRecentActor: + if order == ResultType.OldestActor: sql += """ NATURAL JOIN ( SELECT actor, min(timestamp) AS timestamp - FROM event_view + FROM event_view %s GROUP BY actor) - """ - - if where: + """ % ("WHERE " + where.sql if where.sql else "") + elif where: sql += " WHERE " + where.sql sql += (" ORDER BY timestamp DESC", @@ -367,7 +359,16 @@ class ZeitgeistEngine: " GROUP BY subj_origin ORDER BY timestamp DESC", " GROUP BY subj_origin ORDER BY timestamp ASC", " GROUP BY subj_origin ORDER BY COUNT(subj_origin) DESC, timestamp DESC", - " GROUP BY subj_origin ORDER BY COUNT(subj_origin) ASC, timestamp ASC")[order] + " GROUP BY subj_origin ORDER BY COUNT(subj_origin) ASC, timestamp ASC", + " GROUP BY actor ORDER BY timestamp ASC", + " GROUP BY subj_interpretation ORDER BY timestamp DESC", + " GROUP BY subj_interpretation ORDER BY timestamp ASC", + " GROUP BY subj_interpretation ORDER BY COUNT(subj_interpretation) DESC", + " GROUP BY subj_interpretation ORDER BY COUNT(subj_interpretation) ASC", + " GROUP BY subj_mimetype ORDER BY timestamp DESC", + " GROUP BY subj_mimetype ORDER BY timestamp ASC", + " GROUP BY subj_mimetype ORDER BY COUNT(subj_mimetype) DESC", + " GROUP BY subj_mimetype ORDER BY COUNT(subj_mimetype) ASC")[order] if max_events > 0: sql += " LIMIT %d" % max_events @@ -379,10 +380,7 @@ class ZeitgeistEngine: result = [row[0] for row in result] elif return_mode == 1: log.debug("Found %d events IDs in %fs" % (len(result), time.time()- t)) - result = self.get_events(ids=[row[0] for row in result], sender=sender) - elif return_mode == 2: - log.debug("Found %d (uri,timestamp) tuples in %fs" % (len(result), time.time()- t)) - result = map(lambda row: (row[0], row[1]), result) + result = self.get_events(ids=[row[0] for row in result], sender=sender) else: raise Exception("%d" % return_mode) @@ -394,12 +392,6 @@ class ZeitgeistEngine: def find_events(self, *args): return self._find_events(1, *args) - def __add_window(self, _set, assoc, landmarks, windows): - if _set & landmarks: # intersection != 0 - windows.append(_set) - for i in _set.difference(landmarks): - assoc[i] += 1 - def find_related_uris(self, timerange, event_templates, result_event_templates, result_storage_state, num_results, result_type): """ @@ -411,81 +403,57 @@ class ZeitgeistEngine: and `result_storage_state` are returned. """ - if result_type == 0 or result_type == 1: - + if result_type in (0, 1): + # Instead of using sliding windows we will be using a graph representation + t1 = time.time() - if len(result_event_templates) == 0: - uris = self._find_events(2, timerange, result_event_templates, - result_storage_state, 0, ResultType.LeastRecentEvents) - else: - uris = self._find_events(2, timerange, result_event_templates + event_templates, + # We pick out the ids for relational event so we can set them as roots + # the ids are taken from the events that match the events_templates + ids = self._find_events(0, timerange, event_templates, result_storage_state, 0, ResultType.LeastRecentEvents) - assoc = defaultdict(int) - - landmarks = self._find_events(2, timerange, event_templates, - result_storage_state, 0, 4) - landmarks = set([unicode(event[0]) for event in landmarks]) - - latest_uris = dict(uris) - events = [unicode(u[0]) for u in uris] - - furis = filter(lambda x: x[0] in landmarks, uris) - if len(furis) == 0: - return [] - - _min = min(furis, key=operator.itemgetter(1)) - _max = max(furis, key=operator.itemgetter(1)) - min_index = uris.index(_min) - WINDOW_SIZE - max_index = uris.index(_max) + WINDOW_SIZE - _min = _min[1] - _max = _max[1] + # Pick out the result_ids for the filtered results we would like to take into account + # the ids are taken from the events that match the result_event_templates + # if no result_event_templates are set we consider all results as allowed + result_ids = [] + if len(result_event_templates) > 0: + result_ids = self._find_events(0, timerange, result_event_templates, + result_storage_state, 0, ResultType.LeastRecentEvents) - if min_index < 0: - min_index = 0 - if max_index > len(events): - max_index = -1 - - func = self.__add_window + # From here we create several graphs with the maximum depth of 2 + # and push all the nodes and vertices (events) in one pot together + # FIXME: the depth should be adaptable + pot = [] + for id in ids: + for x in xrange(id-2,id+3): + if len(result_ids) == 0 or x in result_ids: + pot.append(x) - if len(events) == 0 or len(landmarks) == 0: - return [] + # Out of the pot we get all respected events and count which uris occur most + rows = self._cursor.execute(""" + SELECT id, timestamp, subj_uri FROM event_view + WHERE id IN (%s) + """ % ",".join("%d" % id for id in pot)).fetchall() - windows = [] - - if len(events) <= WINDOW_SIZE: - #TODO bug! windows is not defined, seems the algorithm never touches these loop - func(events, assoc, landmarks, windows) - else: - events = events[min_index:max_index] - offset = WINDOW_SIZE/2 - - for i in xrange(offset): - func(set(events[0: offset - i]), assoc, landmarks, - windows) - func(set(events[len(events) - offset + i: len(events)]), - assoc, landmarks, windows) - - it = iter(events) - result = tuple(islice(it, WINDOW_SIZE)) - for elem in it: - result = result[1:] + (elem,) - func(set(result), assoc, landmarks, windows) - - - log.debug("FindRelatedUris: Finished sliding windows in %fs." % \ + subject_uri_counter = defaultdict(int) + latest_uris = defaultdict(int) + for id, timestamp, uri in rows: + if id not in ids: + subject_uri_counter[uri] += 1 + if latest_uris[uri] < timestamp: + latest_uris[uri] = timestamp + + log.debug("FindRelatedUris: Finished ranking subjects %fs." % \ (time.time()-t1)) if result_type == 0: - sets = [[v, k] for k, v in assoc.iteritems()] + sets = subject_uri_counter.iteritems() elif result_type == 1: - sets = [[latest_uris[k], k] for k in assoc] + sets = ((k, latest_uris[k]) for k in subject_uri_counter) - sets.sort(reverse = True) - sets = map(lambda result: result[1], sets[:num_results]) - - return sets + sets = sorted(sets, reverse=True, key=lambda x: x[1])[:num_results] + return map(lambda result: result[0], sets) else: raise NotImplementedError, "Unsupported ResultType." diff --git a/_zeitgeist/engine/notify.py b/_zeitgeist/engine/notify.py index 4ad85142..9b94aa0c 100644 --- a/_zeitgeist/engine/notify.py +++ b/_zeitgeist/engine/notify.py @@ -22,7 +22,6 @@ import logging from zeitgeist.datamodel import TimeRange -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.notify") class _MonitorProxy (dbus.Interface): @@ -226,7 +225,10 @@ class MonitorManager: if matching_events : log.debug("Notifying %s about %s insertions" % (mon, len(matching_events))) - mon.notify_insert(time_range.intersect(mon.time_range), matching_events) + try: + mon.notify_insert(time_range.intersect(mon.time_range), matching_events) + except Exception: + log.exception("Failed to notify monitor: %s" % mon) def notify_delete (self, time_range, event_ids): """ @@ -241,7 +243,10 @@ class MonitorManager: if intersecting_range: log.debug("Notifying %s about %s deletions" % (mon, len(event_ids))) - mon.notify_delete(intersecting_range, event_ids) + try: + mon.notify_delete(intersecting_range, event_ids) + except Exception: + log.exception("Failed to notify monitor: %s" % mon) def _name_owner_changed (self, owner, old, new): """ diff --git a/_zeitgeist/engine/remote.py b/_zeitgeist/engine/remote.py index 10cf19a1..f67cc98e 100644 --- a/_zeitgeist/engine/remote.py +++ b/_zeitgeist/engine/remote.py @@ -42,7 +42,7 @@ class RemoteInterface(SingletonApplication): :const:`org.gnome.zeitgeist.Engine`. """ _dbus_properties = { - "version": property(lambda self: (0, 5, 1)), + "version": property(lambda self: (0, 5, 2)), } # Initialization diff --git a/_zeitgeist/engine/sql.py b/_zeitgeist/engine/sql.py index d20d86ea..1aea03cc 100644 --- a/_zeitgeist/engine/sql.py +++ b/_zeitgeist/engine/sql.py @@ -27,7 +27,6 @@ import os from _zeitgeist.engine import constants -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.sql") TABLE_MAP = { @@ -93,20 +92,20 @@ def _set_schema_version (cursor, schema_name, version): def _do_schema_upgrade (cursor, schema_name, old_version, new_version): """ Try and upgrade schema `schema_name` from version `old_version` to - `new_version`. This is done by checking for an upgrade module named - '_zeitgeist.engine.upgrades.$schema_name_$old_version_$new_version' - and executing the run(cursor) method of that module + `new_version`. This is done by executing a series of upgrade modules + named '_zeitgeist.engine.upgrades.$schema_name_$(i)_$(i+1)' and executing + the run(cursor) method of those modules until new_version is reached """ - # Fire of the right upgrade module - log.info("Upgrading database '%s' from version %s to %s. This may take a while" % - (schema_name, old_version, new_version)) - upgrader_name = "%s_%s_%s" % (schema_name, old_version, new_version) - module = __import__ ("_zeitgeist.engine.upgrades.%s" % upgrader_name) - eval("module.engine.upgrades.%s.run(cursor)" % upgrader_name) - - # Update the schema version - _set_schema_version(cursor, schema_name, new_version) - + for i in xrange(old_version, new_version): + # Fire of the right upgrade module + log.info("Upgrading database '%s' from version %s to %s. This may take a while" % + (schema_name, i, i+1)) + upgrader_name = "%s_%s_%s" % (schema_name, i, i+1) + module = __import__ ("_zeitgeist.engine.upgrades.%s" % upgrader_name) + eval("module.engine.upgrades.%s.run(cursor)" % upgrader_name) + + # Update the schema version + _set_schema_version(cursor, schema_name, i+1) log.info("Upgrade succesful") def _check_core_schema_upgrade (cursor): @@ -114,7 +113,7 @@ def _check_core_schema_upgrade (cursor): # See if we have the right schema version, and try an upgrade if needed core_schema_version = _get_schema_version(cursor, constants.CORE_SCHEMA) if core_schema_version is not None: - if core_schema_version == constants.CORE_SCHEMA_VERSION: + if core_schema_version >= constants.CORE_SCHEMA_VERSION: return True else: try: diff --git a/_zeitgeist/engine/upgrades/core_0_1.py b/_zeitgeist/engine/upgrades/core_0_1.py index e5dd1caf..ae733491 100644 --- a/_zeitgeist/engine/upgrades/core_0_1.py +++ b/_zeitgeist/engine/upgrades/core_0_1.py @@ -3,7 +3,6 @@ import sys import logging import sqlite3 -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.sql") INTERPRETATION_RENAMES = \ diff --git a/_zeitgeist/loggers/Makefile.am b/_zeitgeist/loggers/Makefile.am deleted file mode 100644 index f3617310..00000000 --- a/_zeitgeist/loggers/Makefile.am +++ /dev/null @@ -1,9 +0,0 @@ -SUBDIRS = datasources - -appdir = $(datadir)/zeitgeist/_zeitgeist/loggers/ - -app_PYTHON = \ - __init__.py \ - iso_strptime.py \ - zeitgeist_setup_service.py \ - zeitgeist_base.py diff --git a/_zeitgeist/loggers/__init__.py b/_zeitgeist/loggers/__init__.py deleted file mode 100644 index e69de29b..00000000 --- a/_zeitgeist/loggers/__init__.py +++ /dev/null diff --git a/_zeitgeist/loggers/datasources/Makefile.am b/_zeitgeist/loggers/datasources/Makefile.am deleted file mode 100644 index 8f6c69a7..00000000 --- a/_zeitgeist/loggers/datasources/Makefile.am +++ /dev/null @@ -1,6 +0,0 @@ -appdir = $(datadir)/zeitgeist/_zeitgeist/loggers/datasources - -app_PYTHON = \ - __init__.py \ - _recentmanager.py \ - recent.py diff --git a/_zeitgeist/loggers/datasources/__init__.py b/_zeitgeist/loggers/datasources/__init__.py deleted file mode 100644 index e69de29b..00000000 --- a/_zeitgeist/loggers/datasources/__init__.py +++ /dev/null diff --git a/_zeitgeist/loggers/datasources/_recentmanager.py b/_zeitgeist/loggers/datasources/_recentmanager.py deleted file mode 100644 index e8160396..00000000 --- a/_zeitgeist/loggers/datasources/_recentmanager.py +++ /dev/null @@ -1,146 +0,0 @@ -# -.- coding: utf-8 -.- - -# Zeitgeist -# -# Copyright © 2009 Markus Korn <thekorn@gmx.de> -# Copyright © 2009 Siegfried-Angel Gevatter Pujals <rainct@ubuntu.com> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -import urllib -import gobject -import gio -import os.path -import time -import logging -from xml.dom.minidom import parse as minidom_parse - -from _zeitgeist.loggers.iso_strptime import iso_strptime - -DST = bool(time.mktime(time.gmtime(0))) -log = logging.getLogger("zeitgeist.logger._recentmanager") - -class FileInfo(object): - - @staticmethod - def convert_timestring(time_str): - # My observation is that all times in self.RECENTFILE are in UTC (I might be wrong here) - # so we need to parse the time string into a timestamp - # and correct the result by the timezone difference - try: - timetuple = time.strptime(time_str, "%Y-%m-%dT%H:%M:%SZ") - except ValueError: - timetuple = iso_strptime(time_str.rstrip("Z")).timetuple() - result = int(time.mktime(timetuple)) - if DST: - result -= time.altzone - return result - - def __init__(self, node): - self._uri = node.getAttribute("href") - self._path = "/%s" % self._uri.split("///", 1)[-1] - self._added = self.convert_timestring(node.getAttribute("added")) - self._modified = self.convert_timestring(node.getAttribute("modified")) - self._visited = self.convert_timestring(node.getAttribute("visited")) - - mimetype = node.getElementsByTagNameNS( - "http://www.freedesktop.org/standards/shared-mime-info", - "mime-type") - if not mimetype: - raise ValueError, "Could not find mimetype for item: %s" % self._uri - self._mimetype = mimetype[-1].getAttribute("type") - - applications = node.getElementsByTagNameNS( - "http://www.freedesktop.org/standards/desktop-bookmarks", - "applications") - assert applications - application = applications[0].getElementsByTagNameNS( - "http://www.freedesktop.org/standards/desktop-bookmarks", - "application") - if not application: - raise ValueError, "Could not find application for item: %s" % self._uri - self._application = application[-1].getAttribute("exec").strip("'") - - def get_mime_type(self): - return self._mimetype - - def get_visited(self): - return self._visited - - def get_added(self): - return self._added - - def get_modified(self): - return self._modified - - def get_uri_display(self): - return self._path - - def get_uri(self): - return self._uri - - def get_display_name(self): - return unicode(os.path.basename(urllib.unquote(str(self._path)))) - - def exists(self): - if not self._uri.startswith("file:///"): - return True # Don't check online resources - return gio.File(self._path).get_path() is not None - - def get_private_hint(self): - return False # FIXME: How to get this? - - def last_application(self): - # Not necessary, our get_application_info always returns the info of - # the last application - return "" - - def get_application_info(self, app): - return (self._application, None, None) - -class RecentManager(gobject.GObject): - - RECENTFILE = os.path.expanduser("~/.recently-used.xbel") - - def __init__(self): - super(RecentManager, self).__init__() - if not os.path.exists(self.RECENTFILE): - raise OSError("Can't use alternative RecentManager, '%s' not found" % self.RECENTFILE) - - self._fetching_items = None - file_object = gio.File(self.RECENTFILE) - self.file_monitor = file_object.monitor_file() - self.file_monitor.set_rate_limit(1600) # for to high rates RecentManager - # gets hickup, not sure what's optimal here - self.file_monitor.connect("changed", self._content_changed) - - def _content_changed(self, monitor, fileobj, _, event): - # Only emit the signal if we aren't already parsing RECENTFILE - if not self._fetching_items: - self.emit("changed") - - def get_items(self): - self._fetching_items = True - xml = minidom_parse(self.RECENTFILE) - for bookmark in xml.getElementsByTagName("bookmark"): - yield FileInfo(bookmark) - self._fetching_items = False - - def set_limit(self, limit): - pass - -gobject.type_register(RecentManager) - -gobject.signal_new("changed", RecentManager, - gobject.SIGNAL_RUN_LAST, gobject.TYPE_NONE, ()) diff --git a/_zeitgeist/loggers/datasources/recent.py b/_zeitgeist/loggers/datasources/recent.py deleted file mode 100644 index ff16530e..00000000 --- a/_zeitgeist/loggers/datasources/recent.py +++ /dev/null @@ -1,339 +0,0 @@ -# -.- coding: utf-8 -.- - -# Zeitgeist -# -# Copyright © 2009 Alex Graveley <alex.graveley@beatniksoftewarel.com> -# Copyright © 2009 Markus Korn <thekorn@gmx.de> -# Copyright © 2009 Natan Yellin <aantny@gmail.com> -# Copyright © 2009 Seif Lotfy <seif@lotfy.com> -# Copyright © 2009 Shane Fagan <shanepatrickfagan@yahoo.ie> -# Copyright © 2009-2010 Siegfried-Angel Gevatter Pujals <rainct@ubuntu.com> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -from __future__ import with_statement -import os -import re -import fnmatch -import urllib -import time -import logging -from xdg import BaseDirectory - -from zeitgeist import _config -from zeitgeist.datamodel import Event, Subject, Interpretation, Manifestation, \ - DataSource, get_timestamp_for_now -from _zeitgeist.loggers.zeitgeist_base import DataProvider - -log = logging.getLogger("zeitgeist.logger.datasources.recent") - -try: - import gtk - if gtk.pygtk_version >= (2, 15, 2): - recent_manager = gtk.recent_manager_get_default - else: - from _recentmanager import RecentManager - recent_manager = RecentManager -except ImportError: - log.exception(_("Could not import GTK; data source disabled.")) - enabled = False -else: - enabled = True - -class SimpleMatch(object): - """ Wrapper around fnmatch.fnmatch which allows to define mimetype - patterns by using shell-style wildcards. - """ - - def __init__(self, pattern): - self.__pattern = pattern - - def match(self, text): - return fnmatch.fnmatch(text, self.__pattern) - - def __repr__(self): - return "%s(%r)" %(self.__class__.__name__, self.__pattern) - -DOCUMENT_MIMETYPES = [ - # Covers: - # vnd.corel-draw - # vnd.ms-powerpoint - # vnd.ms-excel - # vnd.oasis.opendocument.* - # vnd.stardivision.* - # vnd.sun.xml.* - SimpleMatch(u"application/vnd.*"), - # Covers: x-applix-word, x-applix-spreadsheet, x-applix-presents - SimpleMatch(u"application/x-applix-*"), - # Covers: x-kword, x-kspread, x-kpresenter, x-killustrator - re.compile(u"application/x-k(word|spread|presenter|illustrator)"), - u"application/ms-powerpoint", - u"application/msword", - u"application/pdf", - u"application/postscript", - u"application/ps", - u"application/rtf", - u"application/x-abiword", - u"application/x-gnucash", - u"application/x-gnumeric", - SimpleMatch(u"application/x-java*"), - SimpleMatch(u"*/x-tex"), - SimpleMatch(u"*/x-latex"), - SimpleMatch(u"*/x-dvi"), - u"text/plain" -] - -IMAGE_MIMETYPES = [ - # Covers: - # vnd.corel-draw - u"application/vnd.corel-draw", - # Covers: x-kword, x-kspread, x-kpresenter, x-killustrator - re.compile(u"application/x-k(word|spread|presenter|illustrator)"), - SimpleMatch(u"image/*"), -] - -AUDIO_MIMETYPES = [ - SimpleMatch(u"audio/*"), - u"application/ogg" -] - -VIDEO_MIMETYPES = [ - SimpleMatch(u"video/*"), - u"application/ogg" -] - -DEVELOPMENT_MIMETYPES = [ - u"application/ecmascript", - u"application/javascript", - u"application/x-csh", - u"application/x-designer", - u"application/x-desktop", - u"application/x-dia-diagram", - u"application/x-fluid", - u"application/x-glade", - u"application/xhtml+xml", - u"application/x-java-archive", - u"application/x-m4", - u"application/xml", - u"application/x-object", - u"application/x-perl", - u"application/x-php", - u"application/x-ruby", - u"application/x-shellscript", - u"application/x-sql", - u"text/css", - u"text/html", - u"text/x-c", - u"text/x-c++", - u"text/x-chdr", - u"text/x-copying", - u"text/x-credits", - u"text/x-csharp", - u"text/x-c++src", - u"text/x-csrc", - u"text/x-dsrc", - u"text/x-eiffel", - u"text/x-gettext-translation", - u"text/x-gettext-translation-template", - u"text/x-haskell", - u"text/x-idl", - u"text/x-java", - u"text/x-lisp", - u"text/x-lua", - u"text/x-makefile", - u"text/x-objcsrc", - u"text/x-ocaml", - u"text/x-pascal", - u"text/x-patch", - u"text/x-python", - u"text/x-sql", - u"text/x-tcl", - u"text/x-troff", - u"text/x-vala", - u"text/x-vhdl", -] - -ALL_MIMETYPES = DOCUMENT_MIMETYPES + IMAGE_MIMETYPES + AUDIO_MIMETYPES + \ - VIDEO_MIMETYPES + DEVELOPMENT_MIMETYPES - -class MimeTypeSet(set): - """ Set which allows to match against a string or an object with a - match() method. - """ - - def __init__(self, *items): - super(MimeTypeSet, self).__init__() - self.__pattern = set() - for item in items: - if isinstance(item, (str, unicode)): - self.add(item) - elif hasattr(item, "match"): - self.__pattern.add(item) - else: - raise ValueError("Bad mimetype '%s'" %item) - - def __contains__(self, mimetype): - result = super(MimeTypeSet, self).__contains__(mimetype) - if not result: - for pattern in self.__pattern: - if pattern.match(mimetype): - return True - return result - - def __len__(self): - return super(MimeTypeSet, self).__len__() + len(self.__pattern) - - def __repr__(self): - items = ", ".join(sorted(map(repr, self | self.__pattern))) - return "%s(%s)" %(self.__class__.__name__, items) - - -class RecentlyUsedManagerGtk(DataProvider): - - FILTERS = { - # dict of name as key and the matching mimetypes as value - # if the value is None this filter matches all mimetypes - "DOCUMENT": MimeTypeSet(*DOCUMENT_MIMETYPES), - "IMAGE": MimeTypeSet(*IMAGE_MIMETYPES), - "AUDIO": MimeTypeSet(*AUDIO_MIMETYPES), - "VIDEO": MimeTypeSet(*VIDEO_MIMETYPES), - "SOURCE_CODE": MimeTypeSet(*DEVELOPMENT_MIMETYPES), - } - - def __init__(self, client): - DataProvider.__init__(self, - unique_id="com.zeitgeist-project,datahub,recent", - name="Recently Used Documents", - description="Logs events from GtkRecentlyUsed", - event_templates=[Event.new_for_values(interpretation=i) for i in ( - Interpretation.CREATE_EVENT, - Interpretation.ACCESS_EVENT, - Interpretation.MODIFY_EVENT - )], - client=client) - self._load_data_sources_registry() - self.recent_manager = recent_manager() - self.recent_manager.set_limit(-1) - self.recent_manager.connect("changed", lambda m: self.emit("reload")) - self.config.connect("configured", lambda m: self.emit("reload")) - - def _load_data_sources_registry(self): - self._ignore_apps = {} - def _data_source_registered(datasource): - for tmpl in datasource[DataSource.EventTemplates]: - actor = tmpl[0][Event.Actor] - if actor: - if not actor in self._ignore_apps: - self._ignore_apps[actor] = set() - interp = tmpl[0][Event.Interpretation] - if interp: - self._ignore_apps[actor].add(interp) - for datasource in self._registry.GetDataSources(): - _data_source_registered(datasource) - self._registry.connect("DataSourceRegistered", _data_source_registered) - - @staticmethod - def _find_desktop_file_for_application(application): - """ Searches for a .desktop file for the given application in - $XDG_DATADIRS and returns the path to the found file. If no file - is found, returns None. - """ - - desktopfiles = \ - list(BaseDirectory.load_data_paths("applications", "%s.desktop" % application)) - if desktopfiles: - return unicode(desktopfiles[0]) - else: - for path in BaseDirectory.load_data_paths("applications"): - for filename in (name for name in os.listdir(path) if name.endswith(".desktop")): - fullname = os.path.join(path, filename) - try: - with open(fullname) as desktopfile: - for line in desktopfile: - if line.startswith("Exec=") and \ - line.split("=", 1)[-1].strip().split()[0] == \ - application: - return unicode(fullname) - except IOError: - pass # file may be a broken symlink (LP: #523761) - except Exception, e: - log.warning('Corrupt .desktop file: %s', fullname) - return None - - def _get_interpretation_for_mimetype(self, mimetype): - matching_filter = None - for filter_name, mimetypes in self.FILTERS.iteritems(): - if mimetype and mimetype in mimetypes: - matching_filter = filter_name - break - if matching_filter: - return getattr(Interpretation, matching_filter).uri - return "" - - def _get_items(self): - # We save the start timestamp to avoid race conditions - last_seen = get_timestamp_for_now() - - events = [] - - for (num, info) in enumerate(self.recent_manager.get_items()): - uri = info.get_uri() - if info.exists() and not info.get_private_hint() and not uri.startswith("file:///tmp/"): - last_application = info.last_application().strip() - application = info.get_application_info(last_application)[0].split()[0] - desktopfile = self._find_desktop_file_for_application(application) - if not desktopfile: - continue - actor = u"application://%s" % os.path.basename(desktopfile) - - subject = Subject.new_for_values( - uri = unicode(uri), - interpretation = self._get_interpretation_for_mimetype( - unicode(info.get_mime_type())), - manifestation = Manifestation.FILE_DATA_OBJECT.uri, - text = info.get_display_name(), - mimetype = unicode(info.get_mime_type()), - origin = uri.rpartition("/")[0] - ) - - times = set() - for meth, interp in ( - (info.get_added, Interpretation.CREATE_EVENT.uri), - (info.get_visited, Interpretation.ACCESS_EVENT.uri), - (info.get_modified, Interpretation.MODIFY_EVENT.uri) - ): - if actor not in self._ignore_apps or \ - (self._ignore_apps[actor] and - interp not in self._ignore_apps[actor]): - times.add((meth() * 1000, interp)) - - is_new = False - for timestamp, use in times: - if timestamp <= self._last_seen: - continue - is_new = True - events.append(Event.new_for_values( - timestamp = timestamp, - interpretation = use, - manifestation = Manifestation.USER_ACTIVITY.uri, - actor = actor, - subjects = [subject] - )) - if num % 50 == 0: - self._process_gobject_events() - self._last_seen = last_seen - return events - -if enabled: - __datasource__ = RecentlyUsedManagerGtk diff --git a/_zeitgeist/loggers/iso_strptime.py b/_zeitgeist/loggers/iso_strptime.py deleted file mode 100644 index 87aa3236..00000000 --- a/_zeitgeist/loggers/iso_strptime.py +++ /dev/null @@ -1,86 +0,0 @@ -# -.- coding: utf-8 -.- - -# This file is part of wadllib. -# -# Copyright © 2009 Canonical Ltd. -# -# wadllib is free software: you can redistribute it and/or modify it under the -# terms of the GNU Lesser General Public License as published by the Free -# Software Foundation, version 3 of the License. -# -# wadllib is distributed in the hope that it will be useful, but WITHOUT ANY -# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more -# details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with wadllib. If not, see <http://www.gnu.org/licenses/>. - -""" -Parser for ISO 8601 time strings -================================ - ->>> d = iso_strptime("2008-01-07T05:30:30.345323+03:00") ->>> d -datetime.datetime(2008, 1, 7, 5, 30, 30, 345323, tzinfo=TimeZone(10800)) ->>> d.timetuple() -(2008, 1, 7, 5, 30, 30, 0, 7, 0) ->>> d.utctimetuple() -(2008, 1, 7, 2, 30, 30, 0, 7, 0) ->>> iso_strptime("2008-01-07T05:30:30.345323-03:00") -datetime.datetime(2008, 1, 7, 5, 30, 30, 345323, tzinfo=TimeZone(-10800)) ->>> iso_strptime("2008-01-07T05:30:30.345323") -datetime.datetime(2008, 1, 7, 5, 30, 30, 345323) ->>> iso_strptime("2008-01-07T05:30:30") -datetime.datetime(2008, 1, 7, 5, 30, 30) ->>> iso_strptime("2008-01-07T05:30:30+02:00") -datetime.datetime(2008, 1, 7, 5, 30, 30, tzinfo=TimeZone(7200)) -""" - -import re -import datetime - -RE_TIME = re.compile(r"""^ - # pattern matching date - (?P<year>\d{4})\-(?P<month>\d{2})\-(?P<day>\d{2}) - # separator - T - # pattern matching time - (?P<hour>\d{2})\:(?P<minutes>\d{2})\:(?P<seconds>\d{2}) - # pattern matching optional microseconds - (\.(?P<microseconds>\d{6})\d*)? - # pattern matching optional timezone offset - (?P<tz_offset>[\-\+]\d{2}\:\d{2})? - $""", re.VERBOSE) - -class TimeZone(datetime.tzinfo): - - def __init__(self, tz_string): - hours, minutes = tz_string.lstrip("-+").split(":") - self.stdoffset = datetime.timedelta(hours=int(hours), - minutes=int(minutes)) - if tz_string.startswith("-"): - self.stdoffset *= -1 - - def __repr__(self): - return "TimeZone(%s)" % ( - self.stdoffset.days*24*60*60 + self.stdoffset.seconds) - - def utcoffset(self, dt): - return self.stdoffset - - def dst(self, dt): - return datetime.timedelta(0) - -def iso_strptime(time_str): - x = RE_TIME.match(time_str) - if not x: - raise ValueError("unable to parse time '%s'" %time_str) - d = datetime.datetime(int(x.group("year")), int(x.group("month")), - int(x.group("day")), int(x.group("hour")), int(x.group("minutes")), - int(x.group("seconds"))) - if x.group("microseconds"): - d = d.replace(microsecond=int(x.group("microseconds"))) - if x.group("tz_offset"): - d = d.replace(tzinfo=TimeZone(x.group("tz_offset"))) - return d diff --git a/_zeitgeist/loggers/zeitgeist_base.py b/_zeitgeist/loggers/zeitgeist_base.py deleted file mode 100644 index ecc15841..00000000 --- a/_zeitgeist/loggers/zeitgeist_base.py +++ /dev/null @@ -1,91 +0,0 @@ -# -.- coding: utf-8 -.- - -# Zeitgeist -# -# Copyright © 2009 Seif Lotfy <seif@lotfy.com> -# Copyright © 2009-2010 Siegfried-Angel Gevatter Pujals <rainct@ubuntu.com> -# Copyright © 2009 Natan Yellin <aantny@gmail.com> -# Copyright © 2009 Alex Graveley <alex@beatniksoftware.com> -# Copyright © 2009 Markus Korn <thekorn@gmx.de> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -from threading import Thread -import gobject -import logging - -from zeitgeist.datamodel import DataSource -from _zeitgeist.loggers.zeitgeist_setup_service import _Configuration, DefaultConfiguration - -class DataProvider(gobject.GObject, Thread): - - __gsignals__ = { - "reload" : (gobject.SIGNAL_RUN_LAST, gobject.TYPE_NONE, ()), - } - - def __init__(self, unique_id, name, description="", event_templates=[], - client=None, config=None): - - # Initialize superclasses - Thread.__init__(self) - gobject.GObject.__init__(self) - - self._name = name - self._client = client - self._ctx = gobject.main_context_default() - - if client: - self._registry = self._client.get_extension("DataSourceRegistry", - "data_source_registry") - try: - self._last_seen = [ds[DataSource.LastSeen] for ds in \ - self._registry.GetDataSources() if \ - ds[DataSource.UniqueId] == unique_id][0] - 1800000 - # We substract 30 minutes to make sure no events get missed. - except IndexError: - self._last_seen = 0 - self._enabled = self._registry.RegisterDataSource(unique_id, name, - description, event_templates) - - if not config: - self.config = DefaultConfiguration(self._name) - else: - if not isinstance(config, _Configuration): - raise TypeError - self.config = config - - def get_name(self): - return self._name - - def get_items(self): - if not self._enabled: - return [] - # FIXME: We need to figure out what to do with this configuration stuff - # Maybe merge it into the DataSource registry so that everyone - # can benefit from it, or just throw it out. - if not self.config.isConfigured() or not self.config.enabled: - logging.warning("'%s' isn't enabled or configured." % \ - self.config.get_internal_name()) - return [] - return self._get_items() - - def _get_items(self): - """ Subclasses should override this to return data. """ - raise NotImplementedError - - def _process_gobject_events(self): - """ Check for pending gobject events. This should be called in some - meaningful place in _get_items on long running updates. """ - while self._ctx.pending(): - self._ctx.iteration() diff --git a/_zeitgeist/loggers/zeitgeist_setup_service.py b/_zeitgeist/loggers/zeitgeist_setup_service.py deleted file mode 100644 index 0c851fe3..00000000 --- a/_zeitgeist/loggers/zeitgeist_setup_service.py +++ /dev/null @@ -1,243 +0,0 @@ -# -.- coding: utf-8 -.- - -# Zeitgeist -# -# Copyright © 2009 Markus Korn <thekorn@gmx.de> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -import dbus -import dbus.service -import gobject -import gconf -import glib -import dbus.mainloop.glib -from ConfigParser import SafeConfigParser -from xdg import BaseDirectory -from StringIO import StringIO - -dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) - -class DataProviderService(dbus.service.Object): - - def __init__(self, datasources, mainloop=None): - bus_name = dbus.service.BusName("org.gnome.zeitgeist.datahub", dbus.SessionBus()) - dbus.service.Object.__init__(self, bus_name, "/org/gnome/zeitgeist/datahub") - self._mainloop = mainloop - self.__datasources = datasources - - @dbus.service.method("org.gnome.zeitgeist.DataHub", - out_signature="as") - def GetDataProviders(self): - return [i.config.get_internal_name() for i in self.__datasources if i.config.has_dbus_service()] - - def needs_setup(self): - return not self.__configuration.isConfigured() - - -class SetupService(dbus.service.Object): - - def __init__(self, datasource, root_config, mainloop=None): - bus_name = dbus.service.BusName("org.gnome.zeitgeist.datahub", dbus.SessionBus()) - dbus.service.Object.__init__(self, - bus_name, "/org/gnome/zeitgeist/datahub/dataprovider/%s" %datasource) - self._mainloop = mainloop - self.__configuration = root_config - if not isinstance(self.__configuration, _Configuration): - raise TypeError - self.__setup_is_running = None - - @dbus.service.method("org.gnome.zeitgeist.DataHub", - in_signature="iss") - def SetConfiguration(self, token, option, value): - if token != self.__setup_is_running: - raise RuntimeError("wrong client") - self.__configuration.set_attribute(option, value) - - @dbus.service.signal("org.gnome.zeitgeist.DataHub") - def NeedsSetup(self): - pass - - @dbus.service.method("org.gnome.zeitgeist.DataHub", - in_signature="i", out_signature="b") - def RequestSetupRun(self, token): - if self.__setup_is_running is None: - self.__setup_is_running = token - return True - else: - raise False - - @dbus.service.method("org.gnome.zeitgeist.DataHub", - out_signature="a(sb)") - def GetOptions(self, token): - if token != self.__setup_is_running: - raise RuntimeError("wrong client") - return self.__configuration.get_options() - - def needs_setup(self): - return not self.__configuration.isConfigured() - - -class _Configuration(gobject.GObject): - - @staticmethod - def like_bool(value): - if isinstance(value, bool): - return value - elif value.lower() in ("true", "1", "on"): - return True - elif value.lower() in ("false", "0", "off"): - return False - else: - raise ValueError - - def __init__(self, internal_name, use_dbus=True, mainloop=None): - gobject.GObject.__init__(self) - self.__required = set() - self.__items = dict() - self.__internal_name = internal_name.replace(" ", "_").lower() - if use_dbus: - self.__dbus_service = SetupService(self.__internal_name, self, mainloop) - else: - self.__dbus_service = None - - def has_dbus_service(self): - return self.__dbus_service is not None - - def get_internal_name(self): - return self.__internal_name - - def add_option(self, name, to_type=str, to_string=str, default=None, - required=True, secret=False): - if name in self.__items: - raise ValueError - if required: - self.__required.add(name) - if to_type is None: - to_type = lambda x: x - self.__items[name] = (to_type(default), (to_type, to_string), secret) - - def __getattr__(self, name): - if not self.isConfigured(): - raise RuntimeError - return self.__items[name][0] - - def get_as_string(self, name): - if not self.isConfigured(): - raise RuntimeError - try: - value, (_, to_string), _ = self.__items[name] - except KeyError: - raise AttributeError - return str(to_string(value)) - - def set_attribute(self, name, value, check_configured=True): - if name not in self.__items: - raise ValueError - _, (to_type, to_string), secret = self.__items[name] - self.__items[name] = (to_type(value), (to_type, to_string), secret) - if name in self.__required: - self.remove_requirement(name) - if check_configured and self.isConfigured(): - glib.idle_add(self.emit, "configured") - - def remove_requirement(self, name): - self.__required.remove(name) - - def add_requirement(self, name): - if not name in self.__items: - raise ValueError - self.__required.add(name) - - def isConfigured(self): - return not self.__required - - def read_config(self, filename, section): - config = SafeConfigParser() - config.readfp(open(filename)) - if config.has_section(section): - for name, value in config.items(section): - self.set_attribute(name, value) - - def dump_config(self, config=None): - section = self.get_internal_name() - if config is None: - config = SafeConfigParser() - try: - config.add_section(section) - except ConfigParser.DuplicateSectionError: - pass - for key, value in self.__items.iteritems(): - value, _, secret = value - if not secret: - config.set(section, key, str(value)) - f = StringIO() - config.write(f) - return f.getvalue() - - def get_requirements(self): - return self.__required - - def get_options(self): - return [(str(key), key in self.__required) for key in self.__items] - - -gobject.signal_new("configured", _Configuration, - gobject.SIGNAL_RUN_LAST, - gobject.TYPE_NONE, - tuple()) - - -class DefaultConfiguration(_Configuration): - - CONFIGFILE = BaseDirectory.load_first_config("zeitgeist", "dataprovider.conf") - DEFAULTS = [ - ("enabled", _Configuration.like_bool, str, True, False), - ] - - def __init__(self, dataprovider): - super(DefaultConfiguration, self).__init__(dataprovider) - for default in self.DEFAULTS: - self.add_option(*default) - if self.CONFIGFILE: - self.read_config(self.CONFIGFILE, self.get_internal_name()) - - def save_config(self): - if self.CONFIGFILE: - config = SafeConfigParser() - config.readfp(open(self.CONFIGFILE)) - self.dump_config(config) - f = StringIO() - config.write(f) - configfile = open(self.CONFIGFILE, "w") - try: - config.write(configfile) - finally: - configfile.close() - -if __name__ == "__main__": - - # TODO: Move this to test/. - - def test(config): - for option, required in config.get_options(): - print option, getattr(config, option) - - dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) - mainloop = gobject.MainLoop() - - config = _Configuration("test", True, mainloop) - config.add_option("enabled", _Configuration.like_bool, default=False) - config.connect("configured", test) - mainloop.run() diff --git a/configure.ac b/configure.ac index 4166ea8f..90949a46 100644 --- a/configure.ac +++ b/configure.ac @@ -1,4 +1,4 @@ -AC_INIT([zeitgeist], [0.5.1]) +AC_INIT([zeitgeist], [0.5.2]) AC_CONFIG_SRCDIR(zeitgeist-daemon.py) AM_INIT_AUTOMAKE([1.9 foreign]) GNOME_COMMON_INIT @@ -26,8 +26,6 @@ AC_CONFIG_FILES([ zeitgeist-daemon.pc zeitgeist/Makefile _zeitgeist/Makefile - _zeitgeist/loggers/Makefile - _zeitgeist/loggers/datasources/Makefile _zeitgeist/engine/Makefile _zeitgeist/engine/extensions/Makefile _zeitgeist/engine/upgrades/Makefile diff --git a/po/POTFILES.in b/po/POTFILES.in index af4dbe4e..20cc7298 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -1,8 +1,4 @@ zeitgeist-daemon.py -zeitgeist-datahub.py zeitgeist/datamodel.py _zeitgeist/singleton.py -_zeitgeist/loggers/zeitgeist_base.py -_zeitgeist/loggers/zeitgeist_setup_service.py -_zeitgeist/loggers/datasources/recent.py _zeitgeist/engine/remote.py diff --git a/po/POTFILES.skip b/po/POTFILES.skip index faa39914..07f231f6 100644 --- a/po/POTFILES.skip +++ b/po/POTFILES.skip @@ -1 +1,3 @@ tools/gtk/zeitgeist-data-sources-gtk.py +tools/cli/zeitgeist-search.py +tools/cli/zeitgeist-stats.py diff --git a/test/blacklist-test.py b/test/blacklist-test.py index 8ad64cfe..b5b06baf 100755 --- a/test/blacklist-test.py +++ b/test/blacklist-test.py @@ -5,7 +5,6 @@ import sys import os import unittest -import dbus sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) from zeitgeist.client import ZeitgeistDBusInterface @@ -19,6 +18,9 @@ class BlacklistTest(RemoteTestCase): self.blacklist = None def setUp(self): + # lazy import to get a chance to use the private bus + import dbus + # We set up the connection lazily in order to wait for the # engine to come up super(BlacklistTest, self).setUp() diff --git a/test/data/twenty_events.js b/test/data/twenty_events.js index 0add8d4c..a8c29651 100644 --- a/test/data/twenty_events.js +++ b/test/data/twenty_events.js @@ -23,10 +23,10 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Video", "manifestation" : "stfu:File", "origin" : "file:///tmp", - "mimetype" : "text/plain", + "mimetype" : "text/booboo", "text" : "this item has not text... rly!", "storage" : "368c991f-8b59-4018-8130-3ce0ec944157" } @@ -42,7 +42,7 @@ "interpretation" : "stfu:Document", "manifestation" : "stfu:File", "origin" : "file:///tmp", - "mimetype" : "text/plain", + "mimetype" : "text/looloo", "text" : "this item has not text... rly!", "storage" : "368c991f-8b59-4018-8130-3ce0ec944157" } @@ -55,10 +55,10 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Video", "manifestation" : "stfu:File", "origin" : "file:///tmp", - "mimetype" : "text/plain", + "mimetype" : "text/looloo", "text" : "this item has not text... rly!", "storage" : "368c991f-8b59-4018-8130-3ce0ec944157" } @@ -103,7 +103,7 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Video", "manifestation" : "stfu:File", "origin" : "file:///tmp", "mimetype" : "text/plain", @@ -122,7 +122,7 @@ "interpretation" : "stfu:Document", "manifestation" : "stfu:File", "origin" : "file:///tmp", - "mimetype" : "text/plain", + "mimetype" : "text/booboo", "text" : "this item has not text... rly!", "storage" : "368c991f-8b59-4018-8130-3ce0ec944157" } @@ -135,7 +135,7 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Music", "manifestation" : "stfu:File", "origin" : "file:///tmp", "mimetype" : "text/plain", @@ -151,7 +151,7 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Music", "manifestation" : "stfu:File", "origin" : "file:///tmp", "mimetype" : "text/plain", @@ -170,7 +170,7 @@ "interpretation" : "stfu:Document", "manifestation" : "stfu:File", "origin" : "file:///tmp", - "mimetype" : "text/plain", + "mimetype" : "text/looloo", "text" : "this item has not text... rly!", "storage" : "368c991f-8b59-4018-8130-3ce0ec944157" } @@ -231,10 +231,10 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Music", "manifestation" : "stfu:File", "origin" : "file:///tmp", - "mimetype" : "text/plain", + "mimetype" : "text/wumbo", "text" : "this item has not text... rly!", "storage" : "368c991f-8b59-4018-8130-3ce0ec944157" } @@ -263,7 +263,7 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Music", "manifestation" : "stfu:File", "origin" : "file:///tmp", "mimetype" : "text/plain", @@ -295,7 +295,7 @@ "subjects" : [ { "uri" : "file:///tmp/foo.txt", - "interpretation" : "stfu:Document", + "interpretation" : "stfu:Something", "manifestation" : "stfu:File", "origin" : "file:///home", "mimetype" : "text/plain", diff --git a/test/engine-extension-test.py b/test/engine-extension-test.py index bf2857ec..b78deee0 100755 --- a/test/engine-extension-test.py +++ b/test/engine-extension-test.py @@ -7,30 +7,23 @@ import os import weakref sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) -import _zeitgeist.engine -from _zeitgeist.engine import constants -from _zeitgeist.engine import get_engine -from _zeitgeist.engine.extension import Extension - import unittest from testutils import import_events -class _Extension1(Extension): - PUBLIC_METHODS = ["return_hallo", "return_engine"] - - def return_hallo(self): - return "Hallo" - - def return_boo(self): - return "boo" - - def return_engine(self): - return self.engine - +Extension = None class _engineTestClass(unittest.TestCase): def setUp (self): + global Extension + + from _zeitgeist.engine import constants + from _zeitgeist.engine import get_engine + + if Extension is None: + from _zeitgeist.engine.extension import Extension as _Extension + Extension = _Extension + constants.DATABASE_FILE = ":memory:" self.save_default_ext = os.environ.get("ZEITGEIST_DEFAULT_EXTENSIONS") self.save_extra_ext = os.environ.get("ZEITGEIST_EXTRA_EXTENSIONS") @@ -39,6 +32,7 @@ class _engineTestClass(unittest.TestCase): self.engine = get_engine() def tearDown (self): + import _zeitgeist.engine if self.save_default_ext is not None: os.environ["ZEITGEIST_DEFAULT_EXTENSIONS"] = self.save_default_ext else: @@ -54,13 +48,27 @@ class _engineTestClass(unittest.TestCase): class TestExtensions(_engineTestClass): def testCreateEngine(self): - engine = get_engine() + + class _Extension1(Extension): + PUBLIC_METHODS = ["return_hallo", "return_engine"] + + def return_hallo(self): + return "Hallo" + + def return_boo(self): + return "boo" + + def return_engine(self): + return self.engine + + engine = self.engine self.assertEqual(len(engine.extensions), 0) self.assertRaises(AttributeError, engine.extensions.__getattr__, "return_hallo") engine.extensions.load(_Extension1) self.assertEqual(engine.extensions.return_hallo(), "Hallo") self.assertRaises(AttributeError, engine.extensions.__getattr__, "return_boo") self.assertEqual(engine.extensions.return_engine(), weakref.proxy(engine)) + class TestExtensionHooks(_engineTestClass): diff --git a/test/engine-test.py b/test/engine-test.py index 42722ad1..994366da 100755 --- a/test/engine-test.py +++ b/test/engine-test.py @@ -6,10 +6,6 @@ import sys import os sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) -import _zeitgeist.engine -from _zeitgeist.engine import constants -from _zeitgeist.engine import get_engine -from _zeitgeist.engine.sql import WhereClause from zeitgeist.datamodel import * from testutils import import_events @@ -41,6 +37,9 @@ def create_test_event_1(): class _engineTestClass(unittest.TestCase): def setUp (self): + from _zeitgeist.engine import constants + from _zeitgeist.engine import get_engine + self.save_default_ext = os.environ.get("ZEITGEIST_DEFAULT_EXTENSIONS") self.save_extra_ext = os.environ.get("ZEITGEIST_EXTRA_EXTENSIONS") os.environ["ZEITGEIST_DEFAULT_EXTENSIONS"] = "" @@ -58,6 +57,7 @@ class _engineTestClass(unittest.TestCase): self.engine = get_engine() def tearDown (self): + import _zeitgeist.engine if self.save_default_ext is not None: os.environ["ZEITGEIST_DEFAULT_EXTENSIONS"] = self.save_default_ext else: @@ -503,195 +503,7 @@ class ZeitgeistEngineTest(_engineTestClass): StorageState.Any, 100, 0,) - self.assertEquals(len(result), 1) - - def testResultTypesMostRecentEvents(self): - import_events("test/data/five_events.js", self.engine) - - # MostRecentEvents - new -> old - ids = self.engine.find_eventids( - TimeRange.always(), [], StorageState.Any, 0, - ResultType.MostRecentEvents) - events = self.engine.get_events(ids) - sorted_event_ids = [ - event.id for event in sorted( - events, cmp=lambda x, y: cmp(int(x.timestamp), int(y.timestamp)), reverse=True - ) - ] - self.assertEquals(ids, sorted_event_ids) - - def testResultTypesLeastRecentEvents(self): - import_events("test/data/five_events.js", self.engine) - - # LeastRecentEvents - old -> new - ids = self.engine.find_eventids( - TimeRange.always(), [], StorageState.Any, 0, - ResultType.LeastRecentEvents) - events = self.engine.get_events(ids) - sorted_event_ids = [ - event.id for event in sorted(events, cmp=lambda x, y: cmp(int(x.timestamp), int(y.timestamp))) - ] - self.assertEquals(ids, sorted_event_ids) - - def testResultTypesMostPopularActor(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularActor) - self.assertEquals([e[0][4] for e in events], ["firefox", "icedove", - "frobnicator"]) - self.assertEquals([e[0][1] for e in events], ["119", "114", "105"]) - - def testResultTypesMostPopularActor2(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange(105,107), [], StorageState.Any, 0, ResultType.MostPopularActor) - self.assertEquals(len(events), 2) - self.assertEquals([e[0][4] for e in events], ["firefox", "frobnicator"]) - self.assertEquals([e[0][1] for e in events], ["107", "105"]) - - def testResultTypesLeastPopularActor(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularActor) - self.assertEquals([e[0][4] for e in events], ["frobnicator", "icedove", - "firefox"]) - self.assertEquals([e[0][1] for e in events], ["105", "114", "119"]) - - def testResultTypesLeastPopularActor2(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange(105,107), [], StorageState.Any, 0, ResultType.LeastPopularActor) - self.assertEquals(len(events), 2) - self.assertEquals([e[0][4] for e in events], ["frobnicator", "firefox"]) - self.assertEquals([e[0][1] for e in events], ["105", "107"]) - - def testResultTypesMostPopularSubject(self): - import_events("test/data/five_events.js", self.engine) - - events = self.engine.find_eventids( - TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularSubjects) - self.assertEquals(events, [3, 5, 4, 1]) - - def testResultTypesLeastPopularSubject(self): - import_events("test/data/five_events.js", self.engine) - - events = self.engine.find_eventids( - TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularSubjects) - self.assertEquals(events, [1, 4, 5, 3]) - - def testResultTypesMostRecentActor(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.MostRecentActor) - self.assertEquals([e[0][1] for e in events], ["119", "114", "105"]) - - def testResultTypesMostRecentActor2(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange(105,107), [], StorageState.Any, 0, ResultType.MostRecentActor) - self.assertEquals([e[0][1] for e in events], ["107", "105"]) - - def testResultTypesLeastRecentActor(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastRecentActor) - self.assertEquals([e[0][1] for e in events], ["100", "101", "105"]) - - def testResultTypesMostPopularOrigin(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularOrigin) - self.assertEquals([e[1][0][3] for e in events], ["file:///tmp", "file:///home", - "file:///etc"]) - self.assertEquals([e[0][1] for e in events], ["116", "118", "119"]) - - def testResultTypesLeastPopularOrigin(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularOrigin) - self.assertEquals([e[1][0][3] for e in events], ["file:///etc", "file:///home", - "file:///tmp"]) - self.assertEquals([e[0][1] for e in events], ["119", "118", "116"]) - - def testResultTypesMostRecentOrigin(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.MostRecentOrigin) - self.assertEquals([e[0][1] for e in events], ["119", "118", "116"]) - - def testResultTypesLeastRecentOrigin(self): - import_events("test/data/twenty_events.js", self.engine) - - events = self.engine.find_events( - TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastRecentOrigin) - self.assertEquals([e[0][1] for e in events], ["116", "118", "119"]) - - def testRelatedForEventsSortRelevancy(self): - import_events("test/data/apriori_events.js", self.engine) - result = self.engine.find_related_uris( - TimeRange.always(), [Event.new_for_values(subject_uri = "i2")], [], - StorageState.Any, 2, 0) - self.assertEquals(result, ["i1", "i3"]) - - def testRelatedForResultTemplateSortRelevancy(self): - import_events("test/data/apriori_events.js", self.engine) - result = self.engine.find_related_uris( - TimeRange.always(), [Event.new_for_values(subject_uri = "i2")], - [Event.new_for_values(subject_uri = "i1")], - StorageState.Any, 2, 0) - self.assertEquals(result, ["i1"]) - - def testRelatedForNoneSortRelevancy(self): - import_events("test/data/apriori_events.js", self.engine) - result = self.engine.find_related_uris( - TimeRange.always(), [], [], - StorageState.Any, 2, 0) - self.assertEquals(result, []) - - def testRelatedForActorSortRelevancy(self): - import_events("test/data/apriori_events.js", self.engine) - event = Event() - event.set_actor("firefox") - result = self.engine.find_related_uris( - TimeRange.always(), [event], [], - StorageState.Any, 2, 0) - logging.debug("************* %s" %result) - self.assertEquals(result, []) - - def testRelatedForEventsSortRecency(self): - import_events("test/data/apriori_events.js", self.engine) - result = self.engine.find_related_uris( - TimeRange.always(), [Event.new_for_values(subject_uri = "i2")], [], - StorageState.Any, 2, 1) - self.assertEquals(result, ["i3", "i1",]) - - def testRelatedForEventsWithManifestation(self): - import_events("test/data/apriori_events.js", self.engine) - result = self.engine.find_related_uris(TimeRange.always(), - [Event.new_for_values(subject_uri = "i4")], - [Event.new_for_values(subject_manifestation="stfu:File")], - StorageState.Any, - 10, 0) - self.assertEquals(result, ["i1", "i3", "i5"]) - - - def testRelatedForMultipleEvents(self): - import_events("test/data/apriori_events.js", self.engine) - result = self.engine.find_related_uris( - TimeRange.always(), [Event.new_for_values(subject_uri = "i1"), - Event.new_for_values(subject_uri = "i4")], [], - StorageState.Any, 2, 0), - self.assertEquals(result, (["i2", "i3", ],)) + self.assertEquals(len(result), 1) def testEventWithBinaryPayload(self): ev = Event() @@ -935,6 +747,8 @@ class ZeitgeistEngineTest(_engineTestClass): self.assertEquals(5, len(ids)) def testWildcardOptimization(self): + from _zeitgeist.engine.sql import WhereClause + cursor = self.engine._cursor strings = [ (u"hällö, I'm gürmen - åge drikker øl - ☠ bug",), @@ -999,6 +813,309 @@ class ZeitgeistEngineTest(_engineTestClass): ).fetchall() ) self.assertEquals(len(cursor.execute(*stm).fetchall()), 1) + + +class FindRelatedUrisTest(_engineTestClass): + + def testRelatedForEventsSortRelevancy(self): + import_events("test/data/apriori_events.js", self.engine) + result = self.engine.find_related_uris( + TimeRange.always(), [Event.new_for_values(subject_uri = "i2")], [], + StorageState.Any, 2, 0) + self.assertEquals(result, ["i1", "i3"]) + + def testRelatedForResultTemplateSortRelevancy(self): + import_events("test/data/apriori_events.js", self.engine) + result = self.engine.find_related_uris( + TimeRange.always(), [Event.new_for_values(subject_uri = "i2")], + [Event.new_for_values(subject_uri = "i1")], + StorageState.Any, 2, 0) + self.assertEquals(result, ["i1"]) + + def testRelatedForNoneSortRelevancy(self): + import_events("test/data/apriori_events.js", self.engine) + result = self.engine.find_related_uris( + TimeRange.always(), [], [], + StorageState.Any, 2, 0) + self.assertEquals(result, []) + + def testRelatedForActorSortRelevancy(self): + import_events("test/data/apriori_events.js", self.engine) + event = Event() + event.set_actor("firefox") + result = self.engine.find_related_uris( + TimeRange.always(), [event], [], + StorageState.Any, 2, 0) + logging.debug("************* %s" %result) + self.assertEquals(result, []) + + def testRelatedForEventsSortRecency(self): + import_events("test/data/apriori_events.js", self.engine) + result = self.engine.find_related_uris( + TimeRange.always(), [Event.new_for_values(subject_uri = "i2")], [], + StorageState.Any, 2, 1) + self.assertEquals(result, ["i3", "i1",]) + + def testRelatedForEventsWithManifestation(self): + import_events("test/data/apriori_events.js", self.engine) + result = self.engine.find_related_uris(TimeRange.always(), + [Event.new_for_values(subject_uri = "i4")], + [Event.new_for_values(subject_manifestation="stfu:File")], + StorageState.Any, + 10, 0) + self.assertEquals(result, ["i1", "i3", "i5"]) + + + def testRelatedForMultipleEvents(self): + import_events("test/data/apriori_events.js", self.engine) + result = self.engine.find_related_uris( + TimeRange.always(), [Event.new_for_values(subject_uri = "i1"), + Event.new_for_values(subject_uri = "i4")], [], + StorageState.Any, 2, 0), + self.assertEquals(result, (["i2", "i3", ],)) + + +class ResultTypeTest(_engineTestClass): + + def testResultTypesMostRecentEvents(self): + import_events("test/data/five_events.js", self.engine) + + # MostRecentEvents - new -> old + ids = self.engine.find_eventids( + TimeRange.always(), [], StorageState.Any, 0, + ResultType.MostRecentEvents) + events = self.engine.get_events(ids) + sorted_event_ids = [ + event.id for event in sorted( + events, cmp=lambda x, y: cmp(int(x.timestamp), int(y.timestamp)), reverse=True + ) + ] + self.assertEquals(ids, sorted_event_ids) + + def testResultTypesLeastRecentEvents(self): + import_events("test/data/five_events.js", self.engine) + + # LeastRecentEvents - old -> new + ids = self.engine.find_eventids( + TimeRange.always(), [], StorageState.Any, 0, + ResultType.LeastRecentEvents) + events = self.engine.get_events(ids) + sorted_event_ids = [ + event.id for event in sorted(events, cmp=lambda x, y: cmp(int(x.timestamp), int(y.timestamp))) + ] + self.assertEquals(ids, sorted_event_ids) + + def testResultTypesMostPopularActor(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularActor) + self.assertEquals([e[0][4] for e in events], ["firefox", "icedove", + "frobnicator"]) + self.assertEquals([e.timestamp for e in events], ["119", "114", "105"]) + + def testResultTypesMostPopularActor2(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange(105,107), [], StorageState.Any, 0, ResultType.MostPopularActor) + self.assertEquals(len(events), 2) + self.assertEquals([e[0][4] for e in events], ["firefox", "frobnicator"]) + self.assertEquals([e.timestamp for e in events], ["107", "105"]) + + def testResultTypesLeastPopularActor(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularActor) + self.assertEquals([e[0][4] for e in events], ["frobnicator", "icedove", + "firefox"]) + self.assertEquals([e.timestamp for e in events], ["105", "114", "119"]) + + def testResultTypesLeastPopularActor2(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange(105,107), [], StorageState.Any, 0, ResultType.LeastPopularActor) + self.assertEquals(len(events), 2) + self.assertEquals([e[0][4] for e in events], ["frobnicator", "firefox"]) + self.assertEquals([e.timestamp for e in events], ["105", "107"]) + + def testResultTypesMostPopularSubject(self): + import_events("test/data/five_events.js", self.engine) + + events = self.engine.find_eventids( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularSubjects) + self.assertEquals(events, [3, 5, 4, 1]) + + def testResultTypesLeastPopularSubject(self): + import_events("test/data/five_events.js", self.engine) + + events = self.engine.find_eventids( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularSubjects) + self.assertEquals(events, [1, 4, 5, 3]) + + def testResultTypesMostRecentActor(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostRecentActor) + self.assertEquals([e.timestamp for e in events], ["119", "114", "105"]) + + def testResultTypesMostRecentActor2(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange(105,107), [], StorageState.Any, 0, ResultType.MostRecentActor) + self.assertEquals([e.timestamp for e in events], ["107", "105"]) + + def testResultTypesOldestActorBug641968(self): + events = [ + Event.new_for_values(timestamp=1, actor="boo", subject_uri="tmp/boo"), + Event.new_for_values(timestamp=2, actor="boo", subject_uri="home/boo"), + Event.new_for_values(timestamp=3, actor="bar", subject_uri="tmp/boo"), + Event.new_for_values(timestamp=4, actor="baz", subject_uri="tmp/boo"), + ] + self.engine.insert_events(events) + + # Get the least recent actors + ids = self.engine.find_eventids(TimeRange.always(), + [], StorageState.Any, 0, ResultType.OldestActor) + self.assertEquals(ids, [1, 3, 4]) + + # Get the least recent actors for "home/boo" + template = Event.new_for_values(subject_uri="home/boo") + ids = self.engine.find_eventids(TimeRange.always(), + [template], StorageState.Any, 0, ResultType.OldestActor) + self.assertEquals(ids, [2]) + + # Let's also try the same with MostRecentActor... Although there + # should be no problem here. + template = Event.new_for_values(subject_uri="home/boo") + ids = self.engine.find_eventids(TimeRange.always(), + [template], StorageState.Any, 0, ResultType.OldestActor) + self.assertEquals(ids, [2]) + + def testResultTypesOldestActor(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), + [Event.new_for_values(subject_manifestation="stfu:File")], + StorageState.Any, 0, ResultType.OldestActor) + self.assertEquals([e.timestamp for e in events], ["100", "101", "105"]) + + def testResultTypesLeastRecentActor(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), + [Event.new_for_values(subject_manifestation="stfu:File")], + StorageState.Any, 0, ResultType.LeastRecentActor) + self.assertEquals([e.timestamp for e in events], ['105', '114', '119']) + + def testResultTypesLeastRecentActor2(self): + # The same test as before, but this time with fewer events so that + # it is actually understandable. + events = [ + Event.new_for_values(timestamp=1, actor="gedit", subject_uri="oldFile"), + Event.new_for_values(timestamp=2, actor="banshee", subject_uri="oldMusic"), + Event.new_for_values(timestamp=3, actor="banshee", subject_uri="newMusic"), + Event.new_for_values(timestamp=4, actor="gedit", subject_uri="newFile"), + ] + self.engine.insert_events(events) + + events = self.engine.find_events(TimeRange.always(), + [], StorageState.Any, 0, ResultType.LeastRecentActor) + self.assertEquals([e.timestamp for e in events], ['3', '4']) + def testResultTypesMostPopularOrigin(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularOrigin) + self.assertEquals([e[1][0][3] for e in events], ["file:///tmp", "file:///home", + "file:///etc"]) + self.assertEquals([e.timestamp for e in events], ["116", "118", "119"]) + + def testResultTypesLeastPopularOrigin(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularOrigin) + self.assertEquals([e[1][0][3] for e in events], ["file:///etc", "file:///home", + "file:///tmp"]) + self.assertEquals([e.timestamp for e in events], ["119", "118", "116"]) + + def testResultTypesMostRecentOrigin(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostRecentOrigin) + self.assertEquals([e.timestamp for e in events], ["119", "118", "116"]) + + def testResultTypesLeastRecentOrigin(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastRecentOrigin) + self.assertEquals([e.timestamp for e in events], ["116", "118", "119"]) + + def testResultTypesMostRecentMimetType(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostRecentMimeType) + self.assertEquals([e.timestamp for e in events], ['119', '114', '110', '107']) + + def testResultTypesLeastRecentMimetType(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastRecentMimeType) + self.assertEquals([e.timestamp for e in events], ['107', '110', '114', '119']) + + def testResultTypesMostPopularMimetType(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularMimeType) + self.assertEquals([e.timestamp for e in events], ['119', '110', '107', '114']) + + def testResultTypesLeastPopularMimetType(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularMimeType) + self.assertEquals([e.timestamp for e in events], ['114', '107', '110', '119']) + + def testResultTypesMostRecentSubjectInterpretation(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostRecentSubjectInterpretation) + self.assertEquals([e.timestamp for e in events], ['119', '118', '116', '106']) + + def testResultTypesLeastRecentSubjectInterpretation(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastRecentSubjectInterpretation) + self.assertEquals([e.timestamp for e in events], ['106', '116', '118', '119']) + + def testResultTypesMostPopularSubjectInterpretation(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.MostPopularSubjectInterpretation) + self.assertEquals([e.timestamp for e in events], ['119', '116', '106', '118']) + + def testResultTypesLeastPopularSubjectInterpretation(self): + import_events("test/data/twenty_events.js", self.engine) + + events = self.engine.find_events( + TimeRange.always(), [], StorageState.Any, 0, ResultType.LeastPopularSubjectInterpretation) + self.assertEquals([e.timestamp for e in events], ['118', '106', '116', '119']) + if __name__ == "__main__": unittest.main() diff --git a/test/loggers-datasources-recent-test.py b/test/loggers-datasources-recent-test.py deleted file mode 100755 index 145b5c33..00000000 --- a/test/loggers-datasources-recent-test.py +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/python - -# Update python path to use local zeitgeist module -import sys -import os -import re -import unittest - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) -from _zeitgeist.loggers.datasources.recent import SimpleMatch, MimeTypeSet - -class SimpleMatchTest(unittest.TestCase): - - def testmatch(self): - self.assertTrue(SimpleMatch("boo/*").match("boo/bar")) - self.assertTrue(SimpleMatch("boo/bar.*").match("boo/bar.foo")) - self.assertFalse(SimpleMatch("boo/bar.*").match("boo/barfoo")) - -class MimeTypeSetTest(unittest.TestCase): - - def testinit(self): - self.assertEquals(repr(MimeTypeSet("boo", "bar", "foo")), "MimeTypeSet('bar', 'boo', 'foo')") - self.assertEquals(repr(MimeTypeSet("boo", "foo", "foo")), "MimeTypeSet('boo', 'foo')") - m = MimeTypeSet("boo", SimpleMatch("bar/*"), re.compile("test.*")) - self.assertEquals(len(m), 3) - self.assertRaises(ValueError, MimeTypeSet, 1) - - def testcontains(self): - m = MimeTypeSet("boo", SimpleMatch("bar/*"), re.compile("test.*")) - self.assertTrue("boo" in m) - self.assertTrue("bar/boo" in m) - self.assertTrue("testboo" in m) - self.assertFalse("boobar" in m) - self.assertFalse("bar" in m) - -if __name__ == '__main__': - unittest.main() diff --git a/test/remote-test.py b/test/remote-test.py index 6216d06f..e9ae984f 100755 --- a/test/remote-test.py +++ b/test/remote-test.py @@ -6,6 +6,8 @@ import sys import logging import signal import time +import tempfile +import shutil from subprocess import Popen, PIPE # DBus setup @@ -18,7 +20,6 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) from zeitgeist.client import ZeitgeistDBusInterface, ZeitgeistClient from zeitgeist.datamodel import (Event, Subject, Interpretation, Manifestation, TimeRange, StorageState) -from _zeitgeist.engine.remote import RemoteInterface import testutils from testutils import parse_events @@ -322,9 +323,31 @@ class ZeitgeistRemoteAPITest(testutils.RemoteTestCase): class ZeitgeistRemoteInterfaceTest(unittest.TestCase): + def setUp(self): + from _zeitgeist import engine + from _zeitgeist.engine import sql, constants + engine._engine = None + sql.unset_cursor() + self.saved_data = { + "datapath": constants.DATA_PATH, + "database": constants.DATABASE_FILE, + "extensions": constants.USER_EXTENSION_PATH, + } + constants.DATA_PATH = tempfile.mkdtemp(prefix="zeitgeist.datapath.") + constants.DATABASE_FILE = ":memory:" + constants.USER_EXTENSION_PATH = os.path.join(constants.DATA_PATH, "extensions") + + def tearDown(self): + from _zeitgeist.engine import constants + shutil.rmtree(constants.DATA_PATH) + constants.DATA_PATH = self.saved_data["datapath"] + constants.DATABASE_FILE = self.saved_data["database"] + constants.USER_EXTENSION_PATH = self.saved_data["extensions"] + def testQuit(self): """calling Quit() on the remote interface should shutdown the engine in a clean way""" + from _zeitgeist.engine.remote import RemoteInterface interface = RemoteInterface() self.assertEquals(interface._engine.is_closed(), False) interface.Quit() @@ -332,17 +355,21 @@ class ZeitgeistRemoteInterfaceTest(unittest.TestCase): class ZeitgeistDaemonTest(unittest.TestCase): + def setUp(self): + self.env = os.environ.copy() + self.datapath = tempfile.mkdtemp(prefix="zeitgeist.datapath.") + self.env.update({ + "ZEITGEIST_DATABASE_PATH": ":memory:", + "ZEITGEIST_DATA_PATH": self.datapath, + }) + + def tearDown(self): + shutil.rmtree(self.datapath) + def testSIGHUP(self): """sending a SIGHUP signal to a running deamon instance results in a clean shutdown""" - daemon = Popen( - ["./zeitgeist-daemon.py", "--no-datahub"], stderr=PIPE, stdout=PIPE - ) - # give the daemon some time to wake up - time.sleep(3) - err = daemon.poll() - if err: - raise RuntimeError("Could not start daemon, got err=%i" % err) + daemon = testutils.RemoteTestCase._safe_start_daemon(env=self.env) os.kill(daemon.pid, signal.SIGHUP) err = daemon.wait() self.assertEqual(err, 0) diff --git a/test/run-all-tests.py b/test/run-all-tests.py index 902f92b0..32565d6e 100755 --- a/test/run-all-tests.py +++ b/test/run-all-tests.py @@ -6,36 +6,110 @@ import unittest import doctest import logging import sys +import tempfile +import shutil from optparse import OptionParser -parser = OptionParser() -parser.add_option("-v", action="count", dest="verbosity") -(options, args) = parser.parse_args() - -if options.verbosity: - # do more fine grained stuff later - # redirect all debugging output to stderr - logging.basicConfig(stream=sys.stderr) -else: - logging.basicConfig(filename="/dev/null") sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) # Find the test/ directory -testdir = os.path.dirname(os.path.abspath(__file__)) -doctests = glob.glob(os.path.join(testdir, "*.rst")) - -# Create a test suite to run all tests -# first, add all doctests -arguments = {"module_relative": False, "globs": {"sys": sys}} -suite = doctest.DocFileSuite(*doctests, **arguments) - -# Add all of the tests from each file that ends with "-test.py" -for fname in os.listdir(testdir): - if fname.endswith("-test.py"): - fname = os.path.basename(fname)[:-3] # Get the filename and chop off ".py" - module = __import__(fname) - suite.addTest(unittest.defaultTestLoader.loadTestsFromModule(module)) - -# Run all of the tests -unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite) +TESTDIR = os.path.dirname(os.path.abspath(__file__)) +DOCTESTS = glob.glob(os.path.join(TESTDIR, "*.rst")) + +def doctest_setup(test): + test._datapath = tempfile.mkdtemp(prefix="zeitgeist.datapath.") + test._env = os.environ.copy() + os.environ.update({ + "ZEITGEIST_DATABASE_PATH": ":memory:", + "ZEITGEIST_DATA_PATH": test._datapath + }) + +def doctest_teardown(test): + shutil.rmtree(test._datapath) + os.environ = test._env + +def iter_tests(suite): + for test in suite: + if isinstance(test, unittest.TestSuite): + for t in iter_tests(test): + yield t + else: + yield test + +def get_test_name(test): + return ".".join((test.__class__.__module__, test.__class__.__name__, test._testMethodName)) + +def load_tests(module, pattern): + suite = unittest.defaultTestLoader.loadTestsFromModule(module) + for test in iter_tests(suite): + name = get_test_name(test) + if pattern is not None: + for p in pattern: + if name.startswith(p): + yield test + break + else: + yield test + +def check_name(filename, pattern): + if pattern is None: + return True + for p in pattern: + if os.path.basename(filename).startswith(p): + return True + return False + +def compile_suite(pattern=None): + # Create a test suite to run all tests + + # first, add all doctests + arguments = { + "module_relative": False, + "globs": {"sys": sys}, + "setUp": doctest_setup, + "tearDown": doctest_teardown, + } + doctests = filter(lambda x: check_name(str(x), pattern), DOCTESTS) + suite = doctest.DocFileSuite(*doctests, **arguments) + + # Add all of the tests from each file that ends with "-test.py" + for fname in os.listdir(TESTDIR): + if fname.endswith("-test.py"): + fname = os.path.basename(fname)[:-3] # Get the filename and chop off ".py" + module = __import__(fname) + tests = list(load_tests(module, pattern)) + suite.addTests(tests) + return suite + +if __name__ == "__main__": + parser = OptionParser() + parser.add_option("-v", action="count", dest="verbosity") + (options, args) = parser.parse_args() + + if options.verbosity: + # do more fine grained stuff later + # redirect all debugging output to stderr + logging.basicConfig(stream=sys.stderr) + else: + logging.basicConfig(filename="/dev/null") + + from testutils import DBusPrivateMessageBus + bus = DBusPrivateMessageBus() + err = bus.run(ignore_errors=True) + if err: + print >> sys.stderr, "*** Failed to setup private bus, error was: %s" %err + else: + print >> sys.stderr, "*** Testsuite is running using a private dbus bus" + config = bus.dbus_config.copy() + config.update({"DISPLAY": bus.DISPLAY, "pid.Xvfb": bus.display.pid}) + print >> sys.stderr, "*** Configuration: %s" %config + try: + os.environ["ZEITGEIST_DEFAULT_EXTENSIONS"] = \ + "_zeitgeist.engine.extensions.blacklist.Blacklist," \ + "_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry" + suite = compile_suite(args or None) + # Run all of the tests + unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite) + finally: + bus.quit(ignore_errors=True) diff --git a/test/sql-test.py b/test/sql-test.py index 4571ddbd..264d38e3 100755 --- a/test/sql-test.py +++ b/test/sql-test.py @@ -23,10 +23,16 @@ import sys, os sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) import unittest -from _zeitgeist.engine.sql import * +WhereClause = None class SQLTest (unittest.TestCase): + def setUp(self): + global WhereClause + if WhereClause is None: + from _zeitgeist.engine.sql import WhereClause as _WhereClause + WhereClause = _WhereClause + def testFlat (self): where = WhereClause(WhereClause.AND) where.add ("foo = %s", 10) diff --git a/test/testutils.py b/test/testutils.py index 7556db07..e565ef9e 100644 --- a/test/testutils.py +++ b/test/testutils.py @@ -3,6 +3,7 @@ # Zeitgeist # # Copyright © 2009 Mikkel Kamstrup Erlandsen <mikkel.kamstrup@gmail.com> +# Copyright © 2009-2010 Markus Korn <thekorn@gmx.de> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as published by @@ -22,6 +23,8 @@ import os import time import sys import signal +import tempfile +import shutil from subprocess import Popen, PIPE # DBus setup @@ -41,8 +44,6 @@ except ImportError: # maybe the user is using python < 2.6 import simplejson as json -from zeitgeist.datamodel import Event, Subject - def dict2event(d): ev = Event() ev[0][Event.Id] = d.get("id", "").encode("UTF-8") @@ -74,7 +75,7 @@ def import_events(path, engine): """ Load a collection of JSON event definitions into 'engine'. Fx: - import_events("test/data/single_event.js", self.engine) + import_events("test/data/single_event.js", self.engine) """ events = parse_events(path) @@ -86,21 +87,50 @@ class RemoteTestCase (unittest.TestCase): remote Zeitgeist process """ + @staticmethod + def _get_pid(matching_string): + p1 = Popen(["ps", "aux"], stdout=PIPE, stderr=PIPE) + p2 = Popen(["grep", matching_string], stdin=p1.stdout, stderr=PIPE, stdout=PIPE) + return p2.communicate()[0] + + @staticmethod + def _safe_start_subprocess(cmd, env, timeout=1, error_callback=None): + """ starts `cmd` in a subprocess and check after `timeout` + if everything goes well""" + process = Popen(cmd, stderr=PIPE, stdout=PIPE, env=env) + # give the process some time to wake up + time.sleep(timeout) + error = process.poll() + if error: + cmd = " ".join(cmd) + error = "'%s' exits with error %i." %(cmd, error) + if error_callback: + error += " *** %s" %error_callback(*process.communicate()) + raise RuntimeError(error) + return process + + @staticmethod + def _safe_start_daemon(env=None, timeout=1): + if env is None: + env = os.environ.copy() + + def error_callback(stdout, stderr): + if "--replace" in stderr: + return "%r | %s" %(stderr, RemoteTestCase._get_pid("zeitgeist-daemon").replace("\n", "|")) + else: + return stderr + + return RemoteTestCase._safe_start_subprocess( + ("./zeitgeist-daemon.py", "--no-datahub"), env, timeout, error_callback + ) + def __init__(self, methodName): super(RemoteTestCase, self).__init__(methodName) self.daemon = None self.client = None def spawn_daemon(self): - os.environ.update({"ZEITGEIST_DATABASE_PATH": ":memory:"}) - self.daemon = Popen( - ["./zeitgeist-daemon.py", "--no-datahub"], stderr=sys.stderr, stdout=sys.stderr - ) - # give the daemon some time to wake up - time.sleep(3) - err = self.daemon.poll() - if err: - raise RuntimeError("Could not start daemon, got err=%i" % err) + self.daemon = self._safe_start_daemon(env=self.env) def kill_daemon(self): os.kill(self.daemon.pid, signal.SIGKILL) @@ -109,6 +139,12 @@ class RemoteTestCase (unittest.TestCase): def setUp(self): assert self.daemon is None assert self.client is None + self.env = os.environ.copy() + self.datapath = tempfile.mkdtemp(prefix="zeitgeist.datapath.") + self.env.update({ + "ZEITGEIST_DATABASE_PATH": ":memory:", + "ZEITGEIST_DATA_PATH": self.datapath, + }) self.spawn_daemon() # hack to clear the state of the interface @@ -119,6 +155,7 @@ class RemoteTestCase (unittest.TestCase): assert self.daemon is not None assert self.client is not None self.kill_daemon() + shutil.rmtree(self.datapath) def insertEventsAndWait(self, events): """ @@ -220,3 +257,49 @@ class RemoteTestCase (unittest.TestCase): num_events=num_events, result_type=result_type) mainloop.run() return result + +class DBusPrivateMessageBus(object): + DISPLAY = ":27" + + def _run(self): + os.environ.update({"DISPLAY": self.DISPLAY}) + devnull = file("/dev/null", "w") + self.display = Popen( + ["Xvfb", self.DISPLAY, "-screen", "0", "1024x768x8"], + stderr=devnull, stdout=devnull + ) + # give the display some time to wake up + time.sleep(1) + err = self.display.poll() + if err: + raise RuntimeError("Could not start Xvfb on display %s, got err=%i" %(self.DISPLAY, err)) + dbus = Popen(["dbus-launch"], stdout=PIPE) + time.sleep(1) + self.dbus_config = dict(l.split("=", 1) for l in dbus.communicate()[0].split("\n") if l) + os.environ.update(self.dbus_config) + + def run(self, ignore_errors=False): + try: + return self._run() + except Exception, e: + if ignore_errors: + return e + raise + + def _quit(self): + os.kill(self.display.pid, signal.SIGKILL) + self.display.wait() + pid = int(self.dbus_config["DBUS_SESSION_BUS_PID"]) + os.kill(pid, signal.SIGKILL) + try: + os.waitpid(pid, 0) + except OSError: + pass + + def quit(self, ignore_errors=False): + try: + return self._quit() + except Exception, e: + if ignore_errors: + return e + raise diff --git a/tools/cli/zeitgeist-integrity-checker.py b/tools/cli/zeitgeist-integrity-checker.py new file mode 100755 index 00000000..2430ef85 --- /dev/null +++ b/tools/cli/zeitgeist-integrity-checker.py @@ -0,0 +1,63 @@ +#! /usr/bin/env python +# -.- coding: utf-8 -.- + +# Zeitgeist +# +# Copyright © 2010 Seif Lotfy <seif@lotfy.com> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +from zeitgeist import _config +_config.setup_path() + +from _zeitgeist.engine.sql import get_default_cursor + +cursor = get_default_cursor() + +# Get all ids from all tables +uris = [id[0] for id in cursor.execute("SELECT id FROM uri").fetchall()] +interpretations = [id[0] for id in cursor.execute("SELECT id FROM interpretation").fetchall()] +manifestations = [id[0] for id in cursor.execute("SELECT id FROM manifestation").fetchall()] +actors = [id[0] for id in cursor.execute("SELECT id FROM actor").fetchall()] +mimetypes = [id[0] for id in cursor.execute("SELECT id FROM mimetype").fetchall()] +texts = [id[0] for id in cursor.execute("SELECT id FROM text").fetchall()] +storages = [id[0] for id in cursor.execute("SELECT id FROM storage").fetchall()] +payloads = [id[0] for id in cursor.execute("SELECT id FROM payload").fetchall()] +events = [event for event in cursor.execute("SELECT * FROM event").fetchall()] + +# Check if each event field if they exist in the respected ids table +# if not add to the respected "failure list" +for event in events: + if not event[2] in interpretations and event[2]: + print "event %i: broken interpretation %s" %(event[0], event[2]) + if not event[3] in manifestations and event[3]: + print "event %i: broken manifestations %s" %(event[0], event[3]) + if not event[4] in actors and event[4]: + print "event %i: broken actor %s" %(event[0], event[4]) + if not event[5] in payloads and event[5]: + print "event %i: broken payload %s" %(event[0], event[5]) + if not event[6] in uris and event[6]: + print "event %i: broken subj_id %s" %(event[0], event[6]) + if not event[7] in interpretations and events[7]: + print "event %i: broken subj_interpretation %s" %(event[0], event[7]) + if not event[8] in manifestations and event[8]: + print "event %i: broken subj_manifestations %s" %(event[0], event[8]) + if not event[9] in uris and event[9]: + print "event %i: broken subj_origin %s" %(event[0], event[9]) + if not event[10] in mimetypes and event[10]: + print "event %i: broken subj_mimetype. %s" %(event[0], event[10]) + if not event[11] in texts and event[11]: + print "event %i: broken subj_text %s" %(event[0], event[11]) + if not event[12] in storages and event[12]: + print "event %i: broken subj_storage %s" %(event[0], event[12]) diff --git a/zeitgeist-daemon.py b/zeitgeist-daemon.py index 5bdf68e8..5c38c624 100755 --- a/zeitgeist-daemon.py +++ b/zeitgeist-daemon.py @@ -4,6 +4,7 @@ # Zeitgeist # # Copyright © 2009 Siegfried-Angel Gevatter Pujals <rainct@ubuntu.com> +# Copyright © 2010 Markus Korn <thekorn@gmx.de> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as published by @@ -21,13 +22,12 @@ import sys import os import gobject -import subprocess import dbus.mainloop.glib import gettext import logging import optparse import signal -from copy import copy +from subprocess import Popen, PIPE # Make sure we can find the private _zeitgeist namespace from zeitgeist import _config @@ -38,84 +38,111 @@ _config.setup_path() from _zeitgeist.engine import constants sys.path.insert(0, constants.USER_EXTENSION_PATH) -gettext.install("zeitgeist", _config.localedir, unicode=1) - -def check_loglevel(option, opt, value): - value = value.upper() - if value in Options.log_levels: - return value - raise optparse.OptionValueError( - "option %s: invalid value: %s" % (opt, value)) +gettext.install("zeitgeist", _config.localedir, unicode=True) class Options(optparse.Option): - TYPES = optparse.Option.TYPES + ("log_levels",) - TYPE_CHECKER = copy(optparse.Option.TYPE_CHECKER) + TYPE_CHECKER = optparse.Option.TYPE_CHECKER.copy() + log_levels = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL") + + def check_loglevel(option, opt, value): + value = value.upper() + if value in Options.log_levels: + return value + raise optparse.OptionValueError( + "option %s: invalid value: %s" % (opt, value)) TYPE_CHECKER["log_levels"] = check_loglevel - log_levels = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL') - -parser = optparse.OptionParser(version = _config.VERSION, option_class=Options) -parser.add_option( - "-r", "--replace", - action = "store_true", default=False, dest = "replace", - help = _("if another Zeitgeist instance is already running, replace it")) -parser.add_option( - "--no-datahub", "--no-passive-loggers", - action = "store_false", default=True, dest = "start_datahub", - help = _("do not start zeitgeist-datahub automatically")) -parser.add_option( - "--log-level", - action = "store", type="log_levels", default="DEBUG", dest="log_level", - help = _("how much information should be printed; possible values:") + \ - " %s" % ', '.join(Options.log_levels)) -parser.add_option( - "--quit", - action = "store_true", default=False, dest = "quit", - help = _("if another Zeitgeist instance is already running, replace it")) -parser.add_option( - "--shell-completion", - action = "store_true", default=False, dest = "shell_completion", - help = optparse.SUPPRESS_HELP) -(_config.options, _config.arguments) = parser.parse_args() - -if _config.options.shell_completion: +def which(executable): + """ helper to get the complete path to an executable """ + p = Popen(["which", str(executable)], stderr=PIPE, stdout=PIPE) + p.wait() + return p.stdout.read().strip() or None + +def parse_commandline(): + parser = optparse.OptionParser(version = _config.VERSION, option_class=Options) + parser.add_option( + "-r", "--replace", + action="store_true", default=False, dest="replace", + help=_("if another Zeitgeist instance is already running, replace it")) + parser.add_option( + "--no-datahub", "--no-passive-loggers", + action="store_false", default=True, dest="start_datahub", + help=_("do not start zeitgeist-datahub automatically")) + parser.add_option( + "--log-level", + action="store", type="log_levels", default="DEBUG", dest="log_level", + help=_("how much information should be printed; possible values:") + \ + " %s" % ", ".join(Options.log_levels)) + parser.add_option( + "--quit", + action="store_true", default=False, dest="quit", + help=_("if another Zeitgeist instance is already running, replace it")) + parser.add_option( + "--shell-completion", + action="store_true", default=False, dest="shell_completion", + help=optparse.SUPPRESS_HELP) + return parser + +def do_shell_completion(parser): options = set() for option in (str(option) for option in parser.option_list): options.update(option.split("/")) - print ' '.join(options) - sys.exit(0) - -logging.basicConfig(level=getattr(logging, _config.options.log_level)) - -from _zeitgeist.engine.remote import RemoteInterface - -dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) -mainloop = gobject.MainLoop() - -try: - interface = RemoteInterface(mainloop = mainloop) -except RuntimeError, e: - logging.error(unicode(e)) - sys.exit(1) - -passive_loggers = os.path.join(_config.bindir, "zeitgeist-datahub.py") -if _config.options.start_datahub: - if os.path.isfile(passive_loggers): - devnull = open(os.devnull, 'w') - subprocess.Popen(passive_loggers, stdin=devnull, stdout=devnull, - stderr=devnull) - del devnull + print " ".join(options) + return 0 + +def setup_interface(): + from _zeitgeist.engine.remote import RemoteInterface + dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) + mainloop = gobject.MainLoop() + return mainloop, RemoteInterface(mainloop = mainloop) + +def start_datahub(): + DATAHUB = "zeitgeist-datahub" + # hide all output of the datahub for now, + # in the future we might want to be more verbose here to make + # debugging easier in case sth. goes wrong with the datahub + devnull = open(os.devnull, "w") + try: + # we assume to find the datahub somewhere in PATH + p = Popen(DATAHUB, stdin=devnull, stdout=devnull, stderr=devnull) + except OSError: + logging.warning("Unable to start the datahub, no binary found") else: - logging.warning( - _("File \"%s\" not found, not starting datahub") % passive_loggers) - -def handle_sighup(signum, frame): - """We are using the SIGHUP signal to shutdown zeitgeist in a clean way""" - logging.info("got SIGHUP signal, shutting down zeitgeist interface") - interface.Quit() -signal.signal(signal.SIGHUP, handle_sighup) - -logging.info(_(u"Starting Zeitgeist service...")) -mainloop.run() + # TODO: delayed check if the datahub is still running after some time + # and not failed because of some error + # tell the user which datahub we are running + logging.debug("Running datahub (%s) with PID=%i" %(which(DATAHUB), p.pid)) + +def setup_handle_sighup(interface): + def handle_sighup(signum, frame): + """We are using the SIGHUP signal to shutdown zeitgeist in a clean way""" + logging.info("got SIGHUP signal, shutting down zeitgeist interface") + interface.Quit() + return handle_sighup + +if __name__ == "__main__": + + parser = parse_commandline() + + _config.options, _config.arguments = parser.parse_args() + if _config.options.shell_completion: + sys.exit(do_shell_completion(parser)) + + logging.basicConfig(level=getattr(logging, _config.options.log_level)) + + try: + mainloop, interface = setup_interface() + except RuntimeError, e: + logging.exception("Failed to setup the RemoteInterface") + sys.exit(1) + + if _config.options.start_datahub: + logging.info("Trying to start the datahub") + start_datahub() + + signal.signal(signal.SIGHUP, setup_handle_sighup(interface)) + + logging.info("Starting Zeitgeist service...") + mainloop.run() diff --git a/zeitgeist-datahub.py b/zeitgeist-datahub.py deleted file mode 100755 index 6a04139b..00000000 --- a/zeitgeist-datahub.py +++ /dev/null @@ -1,135 +0,0 @@ -#! /usr/bin/env python -# -.- coding: utf-8 -.- - -# Zeitgeist -# -# Copyright © 2009 Siegfried-Angel Gevatter Pujals <rainct@ubuntu.com> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -import sys -import os -import glob -import gettext -import logging -import gobject -import dbus.exceptions - -from zeitgeist import _config -_config.setup_path() - -from zeitgeist.client import ZeitgeistDBusInterface -from _zeitgeist.loggers.zeitgeist_setup_service import DataProviderService - -gettext.install("zeitgeist", _config.localedir, unicode=1) -logging.basicConfig(level=logging.DEBUG) - -sys.path.insert(0, _config.datasourcedir) - -class DataHub(gobject.GObject): - - __gsignals__ = { - "reload" : (gobject.SIGNAL_RUN_LAST, gobject.TYPE_NONE, ()), - } - - def __init__(self): - - gobject.GObject.__init__(self) - - self._client = ZeitgeistDBusInterface() - self._client.connect_exit(self._daemon_exit) - - # Load the data sources - self._sources = [] - for datasource_file in glob.glob(_config.datasourcedir + '/*.py'): - if not datasource_file.startswith('_'): - self._load_datasource_file(os.path.basename(datasource_file)) - - # Start by fetch new items from all sources - self._sources_queue = list(self._sources) - if not self._sources_queue: - logging.warning(_("No passive loggers found, bye.")) - sys.exit(1) # Mainloop doesn't exist yet, exit directly - self._db_update_in_progress = True - gobject.idle_add(self._update_db_async) - - for source in self._sources: - source.connect("reload", self._update_db_with_source) - - self._mainloop = gobject.MainLoop() - self.dbus_service = DataProviderService(self._sources, None) - self._mainloop.run() - - def _daemon_exit(self): - self._mainloop.quit() - - def _load_datasource_file(self, datasource_file): - - try: - datasource_object = __import__(datasource_file[:-3]) - except ImportError, err: - logging.exception(_("Could not load file: %s" % datasource_file)) - return False - - if hasattr(datasource_object, "__datasource__"): - objs = datasource_object.__datasource__ - for obj in objs if hasattr(objs, "__iter__") else (objs,): - self._sources.append(obj(self._client)) - - def _update_db_with_source(self, source): - """ - Add new items into the database. This funcion should not be - called directly, but instead activated through the "reload" - signal. - """ - - if not source in self._sources_queue: - self._sources_queue.append(source) - if not self._db_update_in_progress: - self._db_update_in_progress = True - gobject.idle_add(self._update_db_async) - - def _update_db_async(self): - - logging.debug(_("Updating database with new %s items") % \ - self._sources_queue[0].get_name()) - - events = self._sources_queue[0].get_items() - if events: - self._insert_events(self._sources_queue[0].get_name(), events) - - del self._sources_queue[0] - - if len(self._sources_queue) == 0: - self._db_update_in_progress = False - return False # Return False to stop this callback - - # Otherwise, if there are more items in the queue return True so - # that GTK+ will continue to call this function in idle CPU time - return True - - def _insert_events(self, source_name, events): - try: - self._client.InsertEvents(events) - except dbus.exceptions.DBusException, error: - error = error.get_dbus_name() - if error == "org.freedesktop.DBus.Error.ServiceUnknown": - logging.warning( - _("Lost connection to zeitgeist-daemon, terminating.")) - self._daemon_exit() - else: - logging.exception(_("Error logging item from \"%s\": %s" % \ - (source_name, error))) - -datahub = DataHub() diff --git a/zeitgeist/client.py b/zeitgeist/client.py index 768f51bb..e224246b 100644 --- a/zeitgeist/client.py +++ b/zeitgeist/client.py @@ -37,7 +37,6 @@ from zeitgeist.datamodel import (Event, Subject, TimeRange, StorageState, SIG_EVENT = "asaasay" -logging.basicConfig(level=logging.DEBUG) log = logging.getLogger("zeitgeist.client") class _DBusInterface(object): diff --git a/zeitgeist/datamodel.py b/zeitgeist/datamodel.py index 36eb7478..792fbd34 100644 --- a/zeitgeist/datamodel.py +++ b/zeitgeist/datamodel.py @@ -1024,19 +1024,36 @@ class ResultType(object): MostPopularSubjects = enum_factory(("One event for each subject only, " "ordered by the popularity of the subject")) LeastPopularSubjects = enum_factory(("One event for each subject only, " - "ordered ascendingly by popularity")) + "ordered ascendingly by popularity of the subject")) MostPopularActor = enum_factory(("The last event of each different actor," "ordered by the popularity of the actor")) LeastPopularActor = enum_factory(("The last event of each different actor," "ordered ascendingly by the popularity of the actor")) - MostRecentActor = enum_factory(("The last event of each different actor")) - LeastRecentActor = enum_factory(("The first event of each different actor")) + MostRecentActor = enum_factory(("The Actor that has been used to most recently")) + LeastRecentActor = enum_factory(("The Actor that has been used to least recently")) MostRecentOrigin = enum_factory(("The last event of each different origin")) LeastRecentOrigin = enum_factory(("The first event of each different origin")) MostPopularOrigin = enum_factory(("The last event of each different origin," "ordered by the popularity of the origins")) LeastPopularOrigin = enum_factory(("The last event of each different origin," "ordered ascendingly by the popularity of the origin")) + OldestActor = enum_factory(("The first event of each different actor")) + MostRecentSubjectInterpretation = enum_factory(("One event for each subject interpretation only, " + "ordered with the most recent events first")) + LeastRecentSubjectInterpretation = enum_factory(("One event for each subject interpretation only, " + "ordered with the least recent events first")) + MostPopularSubjectInterpretation = enum_factory(("One event for each subject interpretation only, " + "ordered by the popularity of the subject interpretation")) + LeastPopularSubjectInterpretation = enum_factory(("One event for each subject interpretation only, " + "ordered ascendingly by popularity of the subject interpretation")) + MostRecentMimeType = enum_factory(("One event for each mimetype only, " + "ordered with the most recent events first")) + LeastRecentMimeType = enum_factory(("One event for each mimetype only, " + "ordered with the least recent events first")) + MostPopularMimeType = enum_factory(("One event for each mimetype only, " + "ordered by the popularity of the mimetype")) + LeastPopularMimeType = enum_factory(("One event for each mimetype only, " + "ordered ascendingly by popularity of the mimetype")) INTERPRETATION_DOC = \ |