1.5. First own plugin¶
In this chapter we will create our first plugin, register a command, work with csv files and create a thread.
But before we start, let’s take a look what kind of application and plugins we want to build.
1.5.1. Example Application¶
This tutorial uses one single example application, which gets extend by each chapter.
The overall use case of the example application is the monitoring of changes on comma separated files (CSV files).
CSV files represent a table of information and can be read and saved by office tools like Microsoft Excel, Libreoffice Calc an others. They are often used in automation processes and in a lot of cases they are created and updated by scripts.
We want to be able to get notified, if a monitored CSV file gets changed and we also want to know the changes. In later chapters we will also store these changes in a database, so that we know the history of a CSV file. And we will also add some kind of a web interface to watch files and history in the browser.
1.5.1.1. Sequence diagram¶
The following sequence diagram shows how user, application and plugin interacts with each other and what kind of tasks must be done during which step.
But don’t worry, we will implement these features step by step.
1.5.2. Preparation¶
Before we can start coding, we have to create a place for our plugin.
Create a folder csv_watcher_plugin
inside the folder CSV-Manager/csv_manager/plugins/
.
Inside the newly created folder create two new files: __init__.py
and csv_watcher_plugin.py
.
At the end your CSV-Manager should look like this:
CSV-Manager/
├── csv_manager
│ ├── applications
│ │ ├── configuration.py
│ │ ├── csv_manager_app.py
│ │ └── __init__.py
│ ├── patterns
│ │ └── __init__.py
│ ├── plugins
│ │ ├── csv_watcher_plugin # We added this
│ │ │ ├── csv_watcher_plugin.py # We added this
│ │ │ └── __init__.py # We added this
│ │ ├── csv_manager_plugin.py
│ │ └── __init__.py
│ └─ ...
└─ ...
Warning
Be sure your __init__.py
is empty and you haven’t copy and pasted an __init__.py
file from another location,
which has some code in it!
1.5.3. Birth of a plugin¶
Now we can start to write our first plugin. Let’s start with an empty plugin and add needed functions step by step.
Open csv_watcher_plugin.py
and add the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | from groundwork.patterns import GwBasePattern
class CsvWatcherPlugin(GwBasePattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
def activate(self):
pass
def deactivate(self):
pass
|
Each plugin and pattern must inherit from GwBasePattern. If you inherit from another pattern, GwBasePattern gets automatically loaded, because the other pattern already inherits from it [1+4].
There are 4 rules, each plugin must follow:
- A name is available in self.name [9]
- The __init__() routine of patterns gets called [10]
- An activate() routine exists [12]
- A deactivate() routine exists [15]
Rule 1,3 and 4 were checked by GwBasePattern during initialisation. If they are not passed, your plugin can not be used.
A missing call of the __init__() routine of patterns (rule 2) may lead to not correct initialised patterns. For example a database pattern has not initiated its database, because its __init__() routine was not called.
1.5.3.1. Plugin registration¶
Right now, your plugin exists only in a code file and groundwork does not know it.
But we want be able to load our plugins inside a groundwork app by just adding its name into the related configuration parameter.
So we have to tell groundwork or better the used Python environment that there is a new plugin.
This can only be done during installation of the CSV-Manager package. And the installation gets its information from
our setup.py
file.
So open CSV-Manager/setup.py
and take a look on the parameter entry_points
:
entry_points={
'console_scripts': ["csv_manager = "
"csv_manager.applications.csv_manager_app:start_app"],
'groundwork.plugin': ["csv_manager_plugin = "
"csv_manager.plugins.csv_manager_plugin:"
"csv_manager_plugin"],
}
entry_points
is a python dictionary, which has 2 keys: console_scripts
and groundwork.plugin
.
console_scripts
are used to register commands for the command line. In this case, it allows us to use
csv_manager
as command instead of calling the needed python file with
python csv_manager/applications/csv_manager_app
.
groundwork.plugins
is the place where all the magic happens.
It is a python list and there we need to add our plugin.
An entry of this list is a normal string, which must follow the syntax <name> = <packages>:<plugin_class>
.
<name> is not used by groundwork and can be everything.
The needed entry for our plugin is
csv_watcher_plugin = csv_manager.plugins.csv_watcher_plugin.csv_watcher_plugin:CsvWatcherPlugin
.
Let’s add it to our setup.py:
entry_points={
'console_scripts': ["csv_manager = "
"csv_manager.applications.csv_manager_app:start_app"],
'groundwork.plugin': ["csv_manager_plugin = "
"csv_manager.plugins.csv_manager_plugin:"
"csv_manager_plugin", # Do not forget the "," here!
"csv_watcher_plugin = "
"csv_manager.plugins.csv_watcher_plugin.csv_watcher_plugin:CsvWatcherPlugin"
],
}
- After saving
setup.py
we have to reinstall theCSV-Manager
:: >>> cd CSV-Manager >>> pip install -e .
And we also need to add our plugin to the application configuration, so that it gets activated.
Open CSV-Manager/csv_manager/applications/configurations.py
and change the PLUGINS = [...]
line to
PLUGINS = ["csv_manager_plugin", "GwPluginsInfo", "CsvWatcherPlugin"]
1.5.3.2. First run¶
Ok, that’s it. Let’s see if our plugin really gets activated:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | >>> csv_manager
2017-01-14 13:08:14,221 - INFO - Application signals initialised
2017-01-14 13:08:14,407 - INFO - Application commands initialised
2017-01-14 13:08:14,407 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 13:08:14,408 - INFO - Application documents initialised
2017-01-14 13:08:14,408 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 13:08:14,409 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 13:08:14,409 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
Usage: csv_manager [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
hello_world Prints hello world
plugin_list List all plugins
|
Line 7 shows us, that the plugin was found and initiated. And in line 8 we see that it also got activated.
1.5.4. Using commands¶
Now we can start to bring some functionality to our plugin.
At first we should make sure that our functions can be called by the user. For this we need to register a command.
To give our plugin access to command registration, we only need to make sure that our plugin class inherits from
GwCommandsPattern
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | from groundwork.patterns import GwCommandsPattern
class CsvWatcherPlugin(GwCommandsPattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
def activate(self):
pass
def deactivate(self):
pass
|
We have changed line 1 and 3 to use GwCommandsPattern
instead of GwBasePattern
The registration of commands should be done inside the activation()
routine to make sure that our commands are only
available, if our plugin really got activated:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | from groundwork.patterns import GwCommandsPattern
class CsvWatcherPlugin(GwCommandsPattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
def activate(self):
self.commands.register("csv_watch",
"Monitors csv files",
self.csv_watcher_command)
def csv_watcher_command(self):
self.log.info("watcher command called")
def deactivate(self):
pass
|
And again a test:
>>> csv_manager
2017-01-14 15:29:32,742 - INFO - Application signals initialised
2017-01-14 15:29:32,956 - INFO - Application commands initialised
2017-01-14 15:29:32,957 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 15:29:32,957 - INFO - Application documents initialised
2017-01-14 15:29:32,958 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 15:29:32,958 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 15:29:32,959 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
Usage: csv_manager [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
csv_watch Monitors csv files # <--- That's our command.
hello_world Prints hello world
plugin_list List all plugins
>>> csv_manager csv_watch
2017-01-14 15:30:47,952 - INFO - Application signals initialised
2017-01-14 15:30:48,134 - INFO - Application commands initialised
2017-01-14 15:30:48,135 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 15:30:48,135 - INFO - Application documents initialised
2017-01-14 15:30:48,136 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 15:30:48,137 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 15:30:48,137 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 15:30:48,137 - INFO - watcher command called # <--- That's our output
1.5.4.1. Adding a command argument¶
Our command gets called, but we still need the information which file our plugin must monitor. Let’s add an argument to our command:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | from click import Argument
from groundwork.patterns import GwCommandsPattern
class CsvWatcherPlugin(GwCommandsPattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
def activate(self):
# Argument for our command, which stores the csv file path.
path_argument = Argument(("csv_file",),
required=True,
type=str)
self.commands.register("csv_watch",
"Monitors csv files",
self.csv_watcher_command,
params=[path_argument])
def csv_watcher_command(self, csv_file):
self.log.info("watcher command called with csv_path: %s" % csv_file)
def deactivate(self):
pass
|
groundwork uses the library click for handling the command line interface.
Therefore arguments are defined by using the Argument
class from click [1].
In line 16 - 18 we define our argument. It gets a name and is marked as required. We also define the type, so that we can be sure that our function always gets a string.
The self.commands.register()
function has a parameters called params
, which takes a list of
arguments and options [23].
We have also updated our command function to accept the argument as function parameter [25]. The given csv file location will also be used inside our log message [26].
Time for a small test:
>>> csv_manager csv_watch test.csv
2017-01-14 15:51:40,617 - INFO - Application signals initialised
2017-01-14 15:51:40,820 - INFO - Application commands initialised
2017-01-14 15:51:40,821 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 15:51:40,821 - INFO - Application documents initialised
2017-01-14 15:51:40,821 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 15:51:40,822 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 15:51:40,823 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 15:51:40,823 - INFO - watcher command called with csv_path: test.csv # <-- It works!
1.5.5. Handling csv files¶
It’s time to start coding the csv handling part of our plugin. At first we need a way to read the content of a csv file. Luckily Python provides a built-in solution for this: The csv module
But before we start, we need a test.csv
file. Create one in the CSV-Manager
folder
and add the following content:
name,city,phone
Daniel,Munich,123-456
Maria,Cologne,111/222
Richard,Paris,0445-4545-4545
Anabel,London,-
We also need the built-in python libraries csv
for csv handling and os
for checks regarding file existence.
So let’s import them by adding the following lines at the beginning of our file csv_watcher_plugin.py
:
import os
import csv
Now we can write the part for reading the csv file:
1 2 3 4 5 6 7 8 9 10 11 | def csv_watcher_command(self, csv_file):
self.log.info("watcher command called with csv_path: %s" % csv_file)
# Check if the given csv_file really exists
if not os.path.exists(csv_file):
self.log.error("CSV file %s does not exist" % csv_file)
with open(csv_file) as csv_file_object:
reader = csv.DictReader(csv_file_object)
for row in reader:
self.log.info(row)
|
Ok that wasn’t much code and all the magic is done in the lines 8-11. As you can see, we read the csv file and log every single line. Nothing more yet.
And again the test on the command line:
>>> csv_manager csv_watch test.csv
2017-01-14 16:32:35,179 - INFO - Application signals initialised
2017-01-14 16:32:35,361 - INFO - Application commands initialised
2017-01-14 16:32:35,362 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 16:32:35,362 - INFO - Application documents initialised
2017-01-14 16:32:35,363 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 16:32:35,364 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 16:32:35,364 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 16:32:35,364 - INFO - watcher command called with csv_path: test.csv
2017-01-14 16:32:35,365 - INFO - {'name': 'Daniel', 'city': 'Munich', 'phone': '123-456'}
2017-01-14 16:32:35,365 - INFO - {'name': 'Maria', 'city': 'Cologne', 'phone': '111/222'}
2017-01-14 16:32:35,365 - INFO - {'name': 'Richard', 'city': 'Paris', 'phone': '0445-4545-4545'}
2017-01-14 16:32:35,365 - INFO - {'name': 'Anabel', 'city': 'London', 'phone': '-'}
As you can see, each row is a dictionary and the first row of our csv-file was selected to hold the needed dictionary key names.
1.5.5.1. Making csv diffs¶
Our application shall monitor csv files for changes. Therefore it must periodically read a file and compare its
current content with an old, stored content. Let’s add this to our csv_watcher_command
function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | def csv_watcher_command(self, csv_file):
self.log.info("watcher command called with csv_path: %s" % csv_file)
# Check if the given csv_file really exists
if not os.path.exists(csv_file):
self.log.error("CSV file %s does not exist" % csv_file)
# Start with an "empty csv file"
old_content = []
while True:
csv_file_object = open(csv_file)
new_content = list(csv.DictReader(csv_file_object))
if new_content != old_content:
self.log.info("Change detected")
# Check if there are new/changed rows
for row in new_content:
if row not in old_content:
self.log.info("New row: %s" % row)
# Check if old rows are missing
for row in old_content:
if row not in new_content:
self.log.info("Missing row: %s" % row)
# Store the current csv file content as old content
old_content = new_content
csv_file_object.close()
# Wait 2 seconds
time.sleep(2)
|
To allow the easiest way of comparision, the csv content is transformed to a python list, where each list element represents a single row [13]. After that we compare the old and the new list [15].
The code detects changes per row. If one single value has changed, the whole row is detected as “New row” [19-21]. And the old row is detected as “Missing row” [24-26].
Let’s see an output example:
>>> csv_manager csv_watch test.csv
2017-01-14 17:20:17,477 - INFO - Application signals initialised
2017-01-14 17:20:17,660 - INFO - Application commands initialised
2017-01-14 17:20:17,660 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 17:20:17,661 - INFO - Application documents initialised
2017-01-14 17:20:17,661 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 17:20:17,662 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 17:20:17,662 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 17:20:17,663 - INFO - watcher command called with csv_path: test.csv
2017-01-14 17:20:17,663 - INFO - Change detected
2017-01-14 17:20:17,663 - INFO - New row: {'city': 'Munich', 'name': 'Daniel', 'phone': '123-456'}
2017-01-14 17:20:17,663 - INFO - New row: {'city': 'Cologne', 'name': 'Maria', 'phone': '111/222'}
2017-01-14 17:20:17,663 - INFO - New row: {'city': 'Paris', 'name': 'Richard', 'phone': '0445-4545-4545'}
2017-01-14 17:20:17,663 - INFO - New row: {'city': 'London', 'name': 'Anabel', 'phone': '-'}
2017-01-14 17:20:33,682 - INFO - Change detected
2017-01-14 17:20:33,682 - INFO - New row: {'city': 'London', 'name': 'Anabel', 'phone': '777-888888'}
2017-01-14 17:20:33,683 - INFO - Missing row: {'city': 'London', 'name': 'Anabel', 'phone': '-'}
^C
Aborted!
At the beginning the whole csv file content is detected as change, because we compared its content to an empty list. In groundwork and databases we will fix this behavior by using a database to store old csv content.
Another problem is that we are using an infinite loop to check our csv-file. See lines 11 and 34 in the code example
above (while True ...
). Therefore we have to hardly stop our application by pressing Ctrl + C.
Also other code from plugins is blocked, too. So in a complex application nothing would work anymore except our
csv watcher code.
Let’s fix this by using a thread.
1.5.6. Working with threads¶
Threads can be used on a computer to execute something in parallel to the current execution. They are an ideal solution for long running tasks like our csv watcher.
groundwork makes the usage of threads very easy. All we need is the GwThreadPattern
and a python function, which
shall be executed in the new thread.
To see the whole picture, here is the complete code of our plugin CsvWatcherPlugin
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | import os
from click import Argument
import csv
import time
from groundwork.patterns import GwCommandsPattern, GwThreadsPattern
class CsvWatcherPlugin(GwCommandsPattern, GwThreadsPattern):
"""
A plugin for monitoring csv files.
"""
def __init__(self, app, **kwargs):
self.name = "CsvWatcherPlugin"
super().__init__(app, **kwargs)
self.csv_file = None
def activate(self):
# Argument for our command, which stores the csv file path.
path_argument = Argument(("csv_file",),
required=True,
type=str)
self.commands.register("csv_watch",
"Monitors csv files",
self.csv_watcher_command,
params=[path_argument])
def csv_watcher_command(self, csv_file):
# Register thread
self.csv_thread = plugin.threads.register("csv_thread_%s" % csv_file, self._csv_watcher_thread,
"Thread for monitoring a csv file in background")
# Start thread
csv_thread.run()
def csv_watcher_thread(self, plugin):
csv_file = plugin.csv_file
self.log.info("watcher command called with csv_path: %s" % csv_file)
# Check if the given csv_file really exists
if not os.path.exists(csv_file):
self.log.error("CSV file %s does not exist" % csv_file)
# Start with an "empty csv file"
old_content = []
while True:
csv_file_object = open(csv_file)
new_content = list(csv.DictReader(csv_file_object))
if new_content != old_content:
self.log.info("Change detected")
# Check if there are new/changed rows
for row in new_content:
if row not in old_content:
self.log.info("New row: %s" % row)
# Check if old rows are missing
for row in old_content:
if row not in new_content:
self.log.info("Missing row: %s" % row)
# Store the current csv file content as old content
old_content = new_content
csv_file_object.close()
# Wait 2 seconds
time.sleep(2)
def deactivate(self):
pass
|
We have created a new function csv_watcher_thread()
and moved all code from
csv_watcher_command()
to it (see lines 29-72).
csv_watcher_command()
is now responsible for registering and starting the thread [32-36].
It also stores the received csv_file to the plugin class itself, so that the thread has access to it [39].
The thread function csv_watcher_thread()
gets a plugin instance as second parameter when the function is called
by the thread-handler. This plugin instance is the instance which has registered the thread.
As our plugin class has done this, the thread function has now access to all plugin variables and functions.
So also to csv_file
[39].
We still have an infinite loop in the thread. Therefore we still must exit our application with Ctrl + c. But compared to the old version, our watcher is not blocking the rest of our application anymore.
1.5.7. Add interval option¶
Currently our application checks the file every 2 seconds. This value is hard coded and that’s not a good idea. We should allow the user to define the interval. So let’s change the code a little bit and add an optional command option.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | from click import Argument, Option
...
def activate(self):
# Argument for our command, which stores the csv file path.
path_argument = Argument(("csv_file",),
required=True,
type=str)
interval_option = Option(("-i", "--interval"),
type=int,
default=10,
help="Sets the time between two checks in seconds")
self.commands.register("csv_watch",
"Monitors csv files",
self.csv_watcher_command,
params=[path_argument, interval_option])
def csv_watcher_command(self, csv_file, interval=10):
...
def csv_watcher_thread(self, plugin):
...
# Wait x seconds
time.sleep(plugin.csv_interval)
|
Let’s check, if the new option is mentioned in the help text for the command:
>>> csv_manager csv_watch --help
2017-01-14 18:49:03,390 - INFO - Application signals initialised
2017-01-14 18:49:03,595 - INFO - Application commands initialised
2017-01-14 18:49:03,595 - INFO - Plugins initialised: csv_manager_plugin
2017-01-14 18:49:03,595 - INFO - Application documents initialised
2017-01-14 18:49:03,596 - INFO - Plugins initialised: GwPluginsInfo
2017-01-14 18:49:03,596 - INFO - Application threads initialised
2017-01-14 18:49:03,597 - INFO - Plugins initialised: CsvWatcherPlugin
2017-01-14 18:49:03,597 - INFO - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
Usage: csv_manager csv_watch [OPTIONS] CSV_FILE
Monitors csv files
Options:
-i, --interval INTEGER Sets the time between two checks in seconds
--help Show this message and exit.
That’s it, We have created an awesome plugin, which can be started and configured via a command on the command line.
On the next chapter First own pattern we take a look into patterns and make our csv-watcher code reusable for other plugins.