1.6. First own pattern¶
Patterns are used to provide technical, reusable resources to plugins. They are not related to the content of data, which they handle. And they provide no final interfaces to the users. Instead they inject an API into plugins, which inherit from them.
A pattern gets automatically loaded when the first plugin, which inherits from it, gets initiated and activated. And it gets deactivated when the last plugin, which inherit from it, gets deactivated.
For our code example this means that the csv watcher would be the perfect pattern. There are several use cases, where a specific plugin could need the help of a csv watcher pattern. For instance to monitor measurement results or to periodically import data from a tool which exports it’s data to a file.
1.6.1. Preparation¶
Before we can start, let’s create the needed python packages and files.
Create a folder csv_watcher_pattern inside the patterns folder of the csv_manager package.
In the new folder create two additional files: __init__.py and csv_watcher_pattern.py.:
>>> tree
CSV-Manager
├── csv_manager
│ ├── applications
│ │ ├── configuration.py
│ │ ├── csv_manager_app.py
│ │ └── __init__.py
│ ├── patterns
│ │ ├── csv_watcher_pattern
│ │ │ ├── csv_watcher_pattern.py
│ │ │ └── __init__.py
│ │ └── __init__.py
│ └ ...
└ ...
1.6.2. Code the pattern base¶
Open the csv_watcher_pattern.py file and add the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | from groundwork.patterns import GwThreadsPattern
class CsvWatcherPattern(GwThreadsPattern):
def __init__(self, app, **kwargs):
super().__init__(app, **kwargs)
# Adds csv_watcher to each plugin
# This may be done several times. Once for each plugin,
# which inherits from CsvWatcherPattern
self.csv_watcher = CsvWatcherPlugin(plugin=self)
# Adds csv_watcher on application level
# This is done only once for each application
if not hasattr(app, "csv_watcher"):
app.csv_watcher = CsvWatcherApplication()
class CsvWatcherPlugin:
"""
Proxy to the CsvWatcherApplication class.
Responsible for adding the correct plugin context,
if a csv_watcher functions gets called inside a plugin
"""
def __init__(self, plugin):
self._plugin = plugin
self._app = plugin.app
self._watchers = {}
def register(self, csv_file, interval):
self._app.csv_watcher.register(csv_file, interval, self._plugin)
def unregister(self, csv_file):
self._app.csv_watcher.unregister(csv_file, self._plugin)
def get(self, csv_file=None):
self._app.csv_watcher.get(csv_file, self._plugin)
class CsvWatcherApplication:
"""
Main class for handling watchers of csv files.
"""
def __init__(self):
self._watchers = {}
def register(self, csv_file, interval, plugin):
pass
def unregister(self, csv_file, plugin):
pass
def get(self, csv_file=None, plugin=None):
pass
|
Wow, that’s a lot of code for one single step.
But to make our goal clear, we want to be able to register new watchers inside a plugin by using something like
self.csv_watcher.register(). We also want to be able to unregister or get watchers, which were registered
by the current plugin only.
Or in others words: A plugin shall only have access to stuff, which it has registered by itself (called plugin context). That’s already the case for groundwork patterns like commands, documents, signals and more.
On the other hand, there must be a way to get access to all watchers. This should be possible by accessing the data
via the application context. Or more detailed: by accessing it via the app object, which is available in each plugin via
self.app. In our case we would get all watchers by using self.app.csv_watchers.get()
The above code has already implemented this.
We have one class CsvWatcherPlugin [19], which cares about functions called in the plugin context.
And another class CsvWatcherApplication [39], that handles the calls on application level.
To save some code, functions of CsvWatcherPlugin are mostly calling the related function of
CsvWatcherApplication. But they add the current plugin as parameter,
so that the results are filtered for the given plugin.
Example: Lets imagine we are working inside a plugin called “AwesomePlugin” and we register a new watcher via
self.csv_watcher.register("test.csv") [29]. This will call the register function of CsvWatcherPlugin
and inside this function app.csv_watcher.register("test.csv", plugin="AwesomePlugin") [46] is called, which is part
of CsvWatcherApplication.
The pattern is responsible for creating these contexts and handling the correct initialisation [10-16].
As you can see, we will have 3 new functions inside a plugin which inherits from CsvWatcherPattern:
register(), unregister and get()
register() and unregister are used to create new watchers or to delete existing ones.
get() provides acccess to existing watchers.
1.6.3. Moving the csv logic¶
Let’s move our csv logic from the plugin into our pattern.
We use a new class CsvWatcher for all functions and information round about a csv watcher.
So it is very easy for a plugin to request a single watcher and get all needed stuff.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | class CsvWatcher:
def __init__(self, csv_file, interval, description, plugin):
self.csv_file = csv_file
self.interval = interval
self.plugin = plugin
self.description = description
# Register thread
self.csv_thread = plugin.threads.register("csv_thread_%s" % csv_file, self._csv_watcher_thread,
"Thread for monitoring a csv file in background")
self.running = self.csv_thread.running
def run(self):
self.csv_thread.run()
def _csv_watcher_thread(self, plugin):
csv_file = plugin.csv_file
interval = plugin.csv_interval
plugin.log.debug("watcher command called with csv_path: %s and interval: %s" % (csv_file, interval))
# Check if the given csv_file really exists
if not os.path.exists(csv_file):
self.log.error("CSV file %s does not exist" % csv_file)
# Start with an "empty csv file"
old_content = []
while True:
csv_file_object = open(csv_file)
new_content = list(csv.DictReader(csv_file_object))
if new_content != old_content:
plugin.log.debug("Change detected")
# Check if there are new/changed rows
for row in new_content:
if row not in old_content:
plugin.log.debug("New row: %s" % row)
# Check if old rows are missing
for row in old_content:
if row not in new_content:
plugin.log.debug("Missing row: %s" % row)
# Store the current csv file content as old content
old_content = new_content
csv_file_object.close()
# Wait x seconds
time.sleep(interval)
class CsvWatcherExistsException(BaseException):
pass
|
As you can see, we only have made some minor changes.
We added a description parameter, so that a plugin can describe the use case during registration of a watcher.
As we are no longer inside a plugin, self.log will not work. Instead we have to use plugin.log to create log
messages.
We have also added an exception, which gets called, if the given csv file does not exist.
Now let’s take a look into our cleaned plugin class:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | from click import Argument, Option
from groundwork.patterns import GwCommandsPattern
from csv_manager.patterns import CsvWatcherPattern
class CsvWatcherPlugin(GwCommandsPattern, CsvWatcherPattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
self.csv_file = None
self.csv_interval = None
self.watcher_thread = None
def activate(self):
# Argument for our command, which stores the csv file path.
path_argument = Argument(("csv_file",),
required=True,
type=str)
interval_option = Option(("-i", "--interval"),
type=int,
default=10,
help="Sets the time between two checks in seconds")
self.commands.register("csv_watch",
"Monitors csv files",
self.csv_watcher_command,
params=[path_argument, interval_option])
def csv_watcher_command(self, csv_file, interval=10):
# Register thread
self.watcher_thread = self.csv_watcher.register(csv_file, interval, "Watcher for %s" % csv_file)
# Start thread
self.watcher_thread.run()
def deactivate(self):
pass
|
It has become much short and we have removed some import statements.
Instead of GwThreadPattern our plugin class inherits from our new CsvWatcherPattern [6].
Because CsvWatcherPattern inherits itself from GwThreadPattern the thread function calls are
still be available in our plugin class (but we do not need them anymore).
Inside the csv_watcher_command() function we are using our new register() and run() functions to handle
our csv watcher [40+43].
1.6.4. All changed files¶
Before we start testing, let’s take a look into all changed files.
1.6.4.1. csv_watcher_pattern.py¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | #!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import time
import csv
from groundwork.patterns import GwThreadsPattern
from groundwork.util import gw_get
class CsvWatcherPattern(GwThreadsPattern):
def __init__(self, app, **kwargs):
super().__init__(app, **kwargs)
# Adds csv_watcher to each plugin
# This may be done several times. Once for each plugin, which inherits from CsvWatcherPattern
self.csv_watcher = CsvWatcherPlugin(plugin=self)
# Adds csv_watcher on application level
# This is done only once for each application
if not hasattr(app, "csv_watcher"):
app.csv_watcher = CsvWatcherApplication()
class CsvWatcherPlugin:
"""
Proxy to the CsvWatcherApplication class.
Responsible for adding the correct plugin context, if a csv_watcher functions gets called inside a plugin
"""
def __init__(self, plugin):
self._plugin = plugin
self._app = plugin.app
self._watchers = {}
def register(self, csv_file, interval, description):
return self._app.csv_watcher.register(csv_file, interval, description, self._plugin)
def unregister(self, csv_file):
self._app.csv_watcher.unregister(csv_file, self._plugin)
def get(self, csv_file=None):
self._app.csv_watcher.get(csv_file, self._plugin)
class CsvWatcherApplication:
"""
Main class for handling watchers of csv files.
"""
def __init__(self):
self._watchers = {}
def register(self, csv_file, interval, description, plugin):
if csv_file in self._watchers.keys():
raise CsvWatcherExistsException("csv file %s is already registered by %s." %
(csv_file, self._watchers[csv_file].plugin.name))
self._watchers[csv_file] = CsvWatcher(csv_file, interval, description, plugin)
return self._watchers[csv_file]
def unregister(self, csv_file, plugin):
pass
def get(self, csv_file=None, plugin=None):
gw_get(self._watchers, csv_file, plugin)
class CsvWatcher:
def __init__(self, csv_file, interval, description, plugin):
self.csv_file = csv_file
self.interval = interval
self.plugin = plugin
self.description = description
# Register thread
self.csv_thread = plugin.threads.register("csv_thread_%s" % csv_file, self._csv_watcher_thread,
"Thread for monitoring a csv file in background")
self.running = self.csv_thread.running
def run(self):
self.csv_thread.run()
def _csv_watcher_thread(self, plugin):
csv_file = self.csv_file
interval = self.interval
plugin.log.info("watcher command called with csv_path: %s and interval: %s" % (csv_file, interval))
# Check if the given csv_file really exists
if not os.path.exists(csv_file):
plugin.log.error("CSV file %s does not exist" % csv_file)
# Start with an "empty csv file"
old_content = []
while True:
csv_file_object = open(csv_file)
new_content = list(csv.DictReader(csv_file_object))
if new_content != old_content:
plugin.log.info("Change detected")
# Check if there are new/changed rows
for row in new_content:
if row not in old_content:
plugin.log.info("New row: %s" % row)
# Check if old rows are missing
for row in old_content:
if row not in new_content:
plugin.log.info("Missing row: %s" % row)
# Store the current csv file content as old content
old_content = new_content
csv_file_object.close()
# Wait x seconds
time.sleep(interval)
class CsvWatcherExistsException(BaseException):
pass
|
1.6.4.2. csv_watcher_plugin.py¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | #!/usr/bin/env python
# -*- coding: utf-8 -*-
from click import Argument, Option
from groundwork.patterns import GwCommandsPattern
from csv_manager.patterns import CsvWatcherPattern
class CsvWatcherPlugin(GwCommandsPattern, CsvWatcherPattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
self.csv_file = None
self.csv_interval = None
self.watcher_thread = None
def activate(self):
# Argument for our command, which stores the csv file path.
path_argument = Argument(("csv_file",),
required=True,
type=str)
interval_option = Option(("-i", "--interval"),
type=int,
default=10,
help="Sets the time between two checks in seconds")
self.commands.register("csv_watch",
"Monitors csv files",
self.csv_watcher_command,
params=[path_argument, interval_option])
def csv_watcher_command(self, csv_file, interval=10):
# Register thread
self.watcher_thread = self.csv_watcher.register(csv_file, interval, "Watcher for %s" % csv_file)
# Start thread
self.watcher_thread.run()
def deactivate(self):
pass
|
1.6.4.3. patterns/__init__.py¶
We have also imported our pattern inside the __init__.py file of the pattern-package.
With this little change we are able to shorten the import-statement of our pattern to
from csv_manager.patterns import csv_manager_pattern.
1 2 3 | #!/usr/bin/env python
# -*- coding: utf-8 -*-
from .csv_watcher_pattern.csv_watcher_pattern import CsvWatcherPattern
|
1.6.5. Testing¶
Let’s call our csv_manager and see if it works:
>>> csv_manager csv_watch -i 5 test.csv
2017-01-15 10:02:18,232 - INFO - Application signals initialised
2017-01-15 10:02:18,417 - INFO - Application commands initialised
2017-01-15 10:02:18,418 - INFO - Plugins initialised: csv_manager_plugin
2017-01-15 10:02:18,418 - INFO - Application documents initialised
2017-01-15 10:02:18,418 - INFO - Plugins initialised: GwPluginsInfo
2017-01-15 10:02:18,419 - INFO - Application threads initialised
2017-01-15 10:02:18,419 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-15 10:02:18,420 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-15 10:02:18,420 - INFO - watcher command called with csv_path: test.csv and interval: 5
2017-01-15 10:02:18,420 - INFO - Change detected
2017-01-15 10:02:18,420 - INFO - New row: {'city': 'Munich', 'phone': '123-456', 'name': 'Daniel'}
2017-01-15 10:02:18,420 - INFO - New row: {'city': 'Cologne', 'phone': '111/222', 'name': 'Maria'}
2017-01-15 10:02:18,420 - INFO - New row: {'city': 'Paris', 'phone': '0445-4545-45451', 'name': 'Richard'}
2017-01-15 10:02:18,421 - INFO - New row: {'city': 'London', 'phone': '777-888888', 'name': 'Anabel'}
That’s it. You have created an awesome pattern, which every plugin can simply use to register watchers for csv files. And you have updated your plugin to use this pattern.
Sadly there is one ugly problem. Our pattern creates log messages and our plugin gets not informed about changes.
In most cases patterns should not create output to the user. This is the job for the plugin, because only the plugin is specific enough to really know what kind of information the user needs and how this should be presented (log, console, website, e-mail, …).
So in the next chapter Adding signals and receivers we will modify our pattern to send signals instead of writing log messages. And we create a receiver in our plugin, so that the plugin can present the changes to the user.
