6. mqttgateway package

6.1. Warning

As of 24 May 2018, most of the docstrings are obsolete. They will be updated gradually as soon as possible.

6.2. Package contents

The mqttgateway library helps in building gateways between connected devices and MQTT systems.

This package has 4 groups of files:

  • the core of the library made of the modules:

    • start_gateway.py which contains the script for the application initialisation and main loop;
    • mqtt_client.py which defines the child of the MQTT Client class of the paho library, needed to implement a few extra features;
    • mqtt_map.py which defines the internal message class internalMsg and the mapping class msgMap.
  • the utilities used by the core and that are really application agnostic; these are in the modules:

    • app_properties.py, a singleton that holds application wide data like name, directories where to look for files, configuration and log information;
    • init_logger, used by app_properties to initialise the loggers and handlers;
    • load_config, used by app_properties to load the configuration;
    • throttled_exception, an exception class that mutes events if they are too frequent, handy for connection problems happening in fast loops.
  • the dummy interface, an empty interface to test the installation of the library and to be used as a template to write a new interface, and which is made of the modules:

  • various data files:

    • default.conf, the file containing all the configuration options and their default
    • mqtt_map_schema.json, the schema of the mapping files;
    • dummy_map.json and dummy2mqtt.conf, the map and configuration file of the dummy interface.

6.3. Modules

6.4. mqttgateway.app_properties module

Application wide properties.

This module defines a singleton that hides some application-wide properties. As any singleton, any instantiation returns always the same object. This one specifically re-routes the __init__ method to ensure that all variables are only updated at the first instantiation and are never changed again. The attributes are only accessible with getters and there are no setters.

class mqttgateway.app_properties.AppProperties(*args, **kwargs)

Bases: object

Singleton holding application properties.

_init_properties(app_path, app_name=None, parse_dict=None)

Initialisation of the properties.

This is effectively the constructor of the class. At first instantiation of this singleton, the __init__ method points to this method. For the following instantiations, the __init__ method points to the _dummy function.

  • app_path (string) – the full path of the launching script, including filename
  • app_name (string) – the application name if different from the filename
  • parse_dict (dict) – not implemented
get_path(path_given, extension=None, dft_name=None, dft_dirs=None)

Returns the absolute path of a file based on defined rules.

The rules are:

  • the default name is dft_name + extension;
  • the default directories are provided by the dft_dirs argument;
  • file paths can be directory only (ends with a /) and are appended with the default name;
  • file paths can be absolute or relative; absolute start with a / and relative are prepended with the default directory;
  • file paths can be file only (no / whatsoever) and are prepended with the default directory;
  • use forward slashes / in all cases, it should work even for Windows systems;
  • however for Windows systems, use of the drive letter might be an issue and has not been tested.

Currently this method could return a path to a file that does not exist, if a file corresponding to the rules is not found.

  • path_given (string) – any type pf path; see rules
  • extension (string) – the default extension of the file
  • dft_name (string) – the default name to be used, usually the application name
  • dft_dirs (list of strings) – the default directories where to look for the file

the full path of the file found.

Return type:


get_file(path_given, extension=None, dft_name=None, dft_dirs=None)

Returns the content of the file defined by the arguments.

This method uses the get_path() to determine the file sought. All the usual exceptions are raised in case of problems. It is assumed the content of the file is text and that the size is small enough to be returned at once. The arguments are the same as get_path().

  • path_given (string) – any type pf path; see rules
  • extension (string) – the default extension of the file
  • dft_name (string) – the default name to be used, usually the application name
  • dft_dirs (list of strings) – the default directories where to look for the file

the full content of the file.

Return type:


get_jsonfile(path_given, extension=None, dft_name=None, dft_dirs=None)

Returns a dictionary with the content of the JSON file defined by the arguments.

This method uses the get_path() to determine the file sought. All the usual exceptions are raised in case of problems. The arguments are the same as get_path().

  • path_given (string) – any type pf path; see rules
  • extension (string) – the default extension of the file
  • dft_name (string) – the default name to be used, usually the application name
  • dft_dirs (list of strings) – the default directories where to look for the file

the content of the JSON file in dictionary format.

Return type:


load_config(cfg_dflt_string, cfg_filepath)

See loadconfig for documentation.


Creates new handlers from log_data and add the new ones to the log handlers list.

Also updates the registered loggers with the newly created handlers. See method initloghandlers for documentation on log_data format.

Parameters:log_data (string) – see related doc for more info.
Returns:a message indicating what has been done, to be potentially logged.
Return type:string

Register the logger and add the existing handlers to it.

Call this method to add the logger in the registry held in AppProperties. Doing so, the logger will inherit all the pre-defined handlers.

Parameters:logger (logging.Logger object) – the logger to register

Name getter.

Returns:the name of the application.
Return type:string

Directories getter.

The relevant directories of the application are computed and stored at once at the launch of the application. If they have been deleted, moved or their name is changed while the application is running, they will not be valid anymore.

The relevant directories are the current working directory, the directory of the launching script and the directory where the configuration file was found (which could be different from the first 2 because of the option to provide it in the command line).

Returns:a list of full paths of relevant directories for the application.
Return type:list

Command line arguments getter.

Returns:the dictionary returned by parser.parse_args().
Return type:dict

Configuration getter.

Returns:the dictionary returned by ConfigParser.RawConfigParser()
Return type:dict

6.5. mqttgateway.dummy_interface module

The dummy interface class definition. Use it as a template.

This module defines the class dummyInterface that will be instantiated by the module start_gateway.

class mqttgateway.dummy_interface.dummyInterface(params, msglist_in, msglist_out)

Bases: object

An interface that doesn’t do anything but allows to test the installation.

The minimum requirement for the interface class is to define 2 out of 3 possible public methods:

  • the constructor __init__,
  • either the loop method or the loop_start method.
  • params (dictionary of strings) – contains all the options from the configuration file This dictionary is initialised by the [INTERFACE] section in the configuration file. All the options in that section generate an entry in the dictionary. Use this to pass parameters from the configuration file to the interface, for example the name of a port, or the speed of a serial communication.
  • msglist_in (MsgList object) – list of incoming messages in their internal representation.
  • msglist_out (MsgList object) – list of outgoing messages in their internal representation.

The method called periodically by the main loop.

Place here your code to interact with your system.

6.6. mqttgateway.dummy_start module

Launcher script for the dummy gateway.

This is an empty gateway to test the installation setup. It allows to test the loading of the configuration files, the log setup and the basic operation of the core application.

One can also use this as a template. If the name conventions have been respected, just change all occurrences of dummy into the name of your interface.


The entry point for the application

6.7. mqttgateway.init_logger module

Functions to create pre-defined handlers and add them to a logger.

Reminder of attributes for formatting log records (from https://docs.python.org/2/library/logging.html#logrecord-attributes):

Attribute name Format Description
asctime %(asctime)s
Human-readable time when the LogRecord was created.
By default this is of the form ‘2003-07-08 16:49:45,896’ (the numbers after the comma are millisecond portion of the time).
created %(created)f
Time when the LogRecord was created (as returned by
filename %(filename)s
Filename portion of pathname. mynote: = module + ‘.py’ for
python scripts.
funcName %(funcName)s Name of function containing the logging call.
levelname %(levelname)s
Text logging level for the message (‘DEBUG’, ‘INFO’,
levelno %(levelno)s
Numeric logging level for the message (DEBUG, INFO,
lineno %(lineno)d
Source line number where the logging call was issued
(if available).
module %(module)s Module (name portion of filename).
msecs %(msecs)d
Millisecond portion of the time when the LogRecord was
message %(message)s
The logged message, computed as msg % args.
This is set when Formatter.format() is invoked.
name %(name)s Name of the logger used to log the call.
pathname %(pathname)s
Full pathname of the source file where the logging call
was issued (if available).
process %(process)d Process ID (if available).
processName %(processName)s Process name (if available). mynote: is always “MainProcess”.
relativeCreated %(relativeCreated)d
Time in milliseconds when the LogRecord was created,
relative to the time the logging module was loaded.
thread %(thread)d Thread ID (if available).
threadName %(threadName)s Thread name (if available).

The argument datefmt to the Formatter class is of the form: “%Y-%m-%d %H:%M:%S”. The complete set of fields can be found here: time.strftime.

The default (ISO8601) formatter seems to be:


The strings to use to identify the logging levels are defined in the constant _LEVELNAMES.

mqttgateway.init_logger._LEVELNAMES = {'CRITICAL': 50, 'DEBUG': 10, 'ERROR': 40, 'INFO': 20, 'WARN': 30, 'WARNING': 30}

value in the logging library}

Type:Dictionary {“level as string”
mqttgateway.init_logger.initlogger(logger, log_data=None)

Configures a logger with pre-defined handlers based on user-defined configuration data.

Uses initloghandlers() to create the handler, check documentation there for more details on the format of log_data.

The logger level is forced to DEBUG here.

  • logger – the actual logger object to be initialised;
  • log_data (dict) – dictionary of configuration data.

A string made of various lines of messages relating to what handler has been added to the logger. This string can then be logged by the caller or silenced as desired.


Returns a list of handlers based on user-defined configuration data.

The log_data has to be in the form of:

    {'level': xxxx },
    {'level': yyyy,
     'path': zzzz,
     'number': xxxx,
     'size': yyyy},
    {'host': xxxx,
     'port': yyyy,
     'address': zzzz,
     'subject': xxxx }

The following handlers are created and appended to the list returned:

  • the standard ‘Stream’ handler, which will always log level WARN and above to stderr;
  • a console log handler;
  • a rotating file handler that requires a log level, a file path (used as is), the maximum size of the file and the desired number of rotating files;
  • an email handler with the level set to CRITICAL.

The functionality is provided by the standard logging library. Check the documentation for more information on the various parameters.

Parameters:log_data (dict) – dictionary of configuration data
Returns:A pair made of the list of handlers and a message (string), to be logged by the caller or silenced as desired.

6.8. mqttgateway.load_config module

Function to facilitate the loading of configuration parameters.

Based on the standard library ConfigParser.

mqttgateway.load_config.loadconfig(cfg_dflt_path, cfg_filepath)

Load the configuration from a file based on a default one.

This function uses the standard library ConfigParser.

The objective of this function is to provide a mechanism to ensure that all the options that are expected by the application are defined and will not need to be checked throughout the code. To achieve that, a default configuration needs to be provided, represented by the string cfg_dflt_path passed as argument. This string is expected to have all the necessary sections and options that the application will need, with their default values. All options need to be listed there, even the ones that HAVE to be updated and have no default value.

The function loads this default configuration, then checks if the configuration file is available, and if found it grabs only the values from the file that are also present in the default configuration. Anything else in the file is not considered, except for the [INTERFACE] section (see below). The result is a configuration object with all the necessary fields, updated by the values in the configuration file, if present, or with the default values if not. The application can therefore call all fields without checking for their existence.

The exception to the above process in the [INTERFACE] section, which is the section reserved for the developper of the gateway to define its own specific options. The options of this section will be loaded ‘as is’ in the Config object. These options will be sent to the interface through a dedicated dictionary. It is then up to the developper of the interface to check their velidity or provide its own defaults for these options.

Finally, the function updates the option location in the section [CONFIG] with the full path of the configuration file used, so that it can be checked or logged if desired later. This allows to make sure which file has been loaded in case there is some ambiguity. The function also ‘logs’ the error in the ‘error’ option of the same section, if any OS exception occurred while opening or reading the configuration file.

  • cfg_dflt_path (string) – represents the default configuration.
  • cfg_filepath (string) – the path of the configuration file; it is used ‘as is’ and if it is relative there is no guarantee of where it will actually point. It is preferrable to send a valid absolute path.

configparser.ConfigParser object loaded with the options of the configuration.

Return type:


6.9. mqttgateway.mqtt_client module

This is a child class of the MQTT client class of the PAHO library.

It includes the management of reconnection when using only the loop() method, which is not included natively in the current PAHO library.

Notes on MQTT behaviour:

  • if not connected, the loop and publish methods will not do anything, but raise no errors either.
  • the loop method handles always only one message per call.
exception mqttgateway.mqtt_client.connectionError(msg=None)

Bases: mqttgateway.throttled_exception.ThrottledException

Base Exception class for this module, inherits from ThrottledException


Returns a string representing the MQTT message object.

As a reminder, the topic is unicode and the payload is binary.

mqttgateway.mqtt_client._on_connect(client, userdata, flags, return_code)

The MQTT callback when a connection is established.

It sets to True the connected attribute and subscribes to the topics available in the message map.

As a reminder, the flags argument is a dictionary with at least the key session present (with a space!) which will be 1 if the session is already present.

mqttgateway.mqtt_client._on_subscribe(client, userdata, mid, granted_qos)

The MQTT callback when a subscription is completed.

Only implemented for debug purposes.

mqttgateway.mqtt_client._on_disconnect(client, userdata, return_code)

The MQTT callback when a disconnection occurs.

It sets to False the mg_connected attribute.

mqttgateway.mqtt_client._on_message(client, userdata, mqtt_msg)

The MQTT callback when a message is received from the MQTT broker.

The message (topic and payload) is mapped into its internal representation and then appended to the incoming message list for the gateway interface to execute it later.

class mqttgateway.mqtt_client.mgClient(host='localhost', port=1883, keepalive=60, client_id='', on_msg_func=None, topics=None, userdata=None)

Bases: paho.mqtt.client.Client

Class representing the MQTT connection. mg means MqttGateway.

Inheritance issues:
The MQTT paho library sets quite a few attributes in the Client class. They all start with an underscore and have standard names (_host, _port,…). Also, some high level methods are used extensively in the paho library itself, in particular the connect() method. Overriding them is dangerous. That is why all the homonymous attributes and methods here have an mg_ prepended to avoid these problems.
  • host (string) – a valid host address for the MQTT broker (excluding port)
  • port (int) – a valid port for the MQTT broker
  • keepalive (int) – see PAHO documentation
  • client_id (string) – the name (usually the application name) to send to the MQTT broker
  • on_msg_func (function) – function to call during on_message()
  • topics (list of strings) – e.g.[‘home/audiovideo/#’, ‘home/lighting/#’]
  • userdata (object) – any object that will be passed to the call-backs

Method to inhibit the connection test during the lag.

One of the feature added by this class over the standard PAHO class is the possibility to reconnect when disconnected while using only the loop() method. In order to achieve this, the connection is checked regularly. At the very beginning of the connection though, there is the possibility of a race condition when testing the connection state too soon after requesting it. This happens if the on_connect call-back is not called fast enough by the PAHO library and the main loop tests the connection state before that call-back has had the time to set the state to connected. As a consequence the automatic reconnection feature gets triggered while a connection is already under way, and the connection process gets jammed with the broker. That’s why we need to leave a little lag before testing the connection. This is done with the function variable lag_test, which is assigned to this function (lag_end) at connection, and switched to a dummy lambda after the lag has passed.


Resets the lag feature for a new connection request.


Sets up the lag feature on top of the parent connect method.

See lag_end for more information on the lag feature.


Sets up the lag feature on top of the parent method.


Implements automatic reconnection on top of the parent loop method.

The use of the method/attribute lag_test() is to avoid having to test the lag forever once the connection is established. Once the lag is finished, this method gets replaced by a simple lambda, which hopefully is much faster than calling the time library and doing a comparison.

6.10. mqttgateway.mqtt_map module

This module is the bridge between the internal and the MQTT representation of messages.

As a reminder, we define the MQTT syntax as follows:

  • topic:

    root/function/gateway/location/device/sender/type-{C or S}
  • payload: action or status, in plain text or in a json string like {key1:value1,key2:value2,..}

class mqttgateway.mqtt_map.internalMsg(iscmd=False, function=None, gateway=None, location=None, device=None, sender=None, action=None, arguments=None)

Bases: object

Defines all the characteristics of an internal message.

Note about the behaviour of None:
a characteristic set to None and one set to an empty string are considered the same, and they both mean a non existent or missing value. It could be interesting to differentiate between then at a later stage as, for example, an empty string could still be mapped to an existing internal value, as if it was a default, but that is not the case here. Therefore None values are always converted to empty strings.

TODO: implement smart retrieval of arguments with key checks

  • iscmd (bool) – Indicates if the message is a command (True) or a status (False)
  • function (string) – internal representation of function
  • gateway (string) – internal representation of gateway
  • location (string) – internal representation of location
  • device (string) – internal representation of device
  • sender (string) – internal representation of sender
  • action (string) – internal representation of action
  • arguments (dictionary of strings) – all values should be assumed to be strings

Clears all content of the message.


Creates a copy of the message.

argument(arg, raises=False, default=None)

Return the argument if found in the arguments dictionary.

reply(response, reason)

Formats the message to be sent as a reply to an existing command

This method is supposed to be used with an existing message that has been received. Using this method for all replies guarantees a consistent syntax for replies.

  • response (string) – code or abbreviation for response, e.g. OK```or ``ERROR
  • reason (string) – longer description of the responses
class mqttgateway.mqtt_map.MsgList

Bases: Queue.Queue, object

Message list to communicate between the library and the interface.

Defined as a Queue list in case the library is used in multi-threading mode.

The methods are called push and pull in order to differentiate them from the usual names (put, get, append, pop, …).

push(item, block=True, timeout=None)

Pushes the item at the end of the list.

Equivalent to append or put in other list implementations. The block and timeout arguments have the same meaning as in the Queue library.

  • item (object) – the object to push in the list
  • block (boolean) – in case the list is full
  • timeout (float) – wait time if block == True
pull(block=False, timeout=None)

Pull the first item from the list.

Equivalent to pop or get in other list implementations. The block and timeout arguments have the same meaning as in the Queue library.

  • block (boolean) – in case the list is empty
  • timeout (float) – wait time if block == True
class mqttgateway.mqtt_map.mappedTokens(function, gateway, location, device, sender, action, argkey, argvalue)

Bases: tuple

Tokens representing a message that can be mapped.


Return a new OrderedDict which maps field names to their values

_fields = ('function', 'gateway', 'location', 'device', 'sender', 'action', 'argkey', 'argvalue')
classmethod _make(iterable, new=<built-in method __new__ of type object>, len=<built-in function len>)

Make a new mappedTokens object from a sequence or iterable


Return a new mappedTokens object replacing specified fields with new values


Alias for field number 5


Alias for field number 6


Alias for field number 7


Alias for field number 3


Alias for field number 0


Alias for field number 1


Alias for field number 2


Alias for field number 4

mqttgateway.mqtt_map.NO_MAP = {'action': {'maptype': 'none'}, 'argkey': {'maptype': 'none'}, 'argvalue': {'maptype': 'none'}, 'device': {'maptype': 'none'}, 'function': {'maptype': 'none'}, 'gateway': {'maptype': 'none'}, 'location': {'maptype': 'none'}, 'root': '', 'sender': {'maptype': 'none'}, 'topics': []}

Default map, with no mapping at all.

class mqttgateway.mqtt_map.msgMap(jsondict=None)

Bases: object

Contains the mapping data and the conversion methods.

The mapping data is read from a JSON style dictionary. To access the maps use:

mqtt_token = maps.*field*.i2m(internal_token)


mqtt_token = maps.gateway.i2m(internal_token)
Parameters:jsondict (dictionary) – contains the map data in the agreed format; if None, the NO_MAP structure is used.
class tokenMap(maptype, mapdict=None)

Bases: object

Represents the mapping for a given token or characteristic.

Each instantiation of this class represent the mapping for a given token, and contains the type of mapping, the mapping dictionary if available, and the methods to convert the keywords back and forth between MQTT and internal representation.

The mapping dictionary passed as argument has the internal keywords as keys and as value a list of corresponding MQTT keywords. Only the first of the list will be used for the reverse dictionary, the other MQTT keywords being ‘aliases’.

  • maptype (string) – type of map, should be either ‘strict’. ‘loose’ or ‘none’
  • mapdict (dictionary) – dictionary representing the mapping

Generic method converting an MQTT token into an internal characteristic.


Generic method converting an internal characteristic into an MQTT token.

static _mapnone(token, dico)

Returns the argument unchanged.

  • token (string) – the token to convert
  • dico (dictionary) – the mapping dictionary to use for the conversion, if needed

converted token

Return type:


static _maploose(token, dico)

Returns the argument converted if in dictionary, unchanged otherwise.

If token is None, it is always converted in an empty string.

  • token (string) – the token to convert
  • dico (dictionary) – the mapping dictionary to use for the conversion, if needed

converted token

Return type:


static _mapstrict(token, dico)

Returns the argument converted if in dictionary, raises exception otherwise.

If token is None, it is always converted in an empty string. An empty string is kept as an empty string, even if not in the dictionary.

  • token (string) – the token to convert
  • dico (dictionary) – the mapping dictionary to use for the conversion, if needed

converted token

Return type:



Getter for the _sender attribute.


Converts the MQTT message into an internal one.

Parameters:mqtt_msg (mqtt.MQTTMessage) – a MQTT message.
Returns:the conversion of the MQTT message
Return type:internalMsg object
Raises:ValueError – in case of bad MQTT syntax or unrecognised map elements

Converts an internal message into a MQTT one.

Parameters:internal_msg (internalMsg) – the message to convert
Returns:a full MQTT message where topic syntax is root/function/gateway/location/device/sender/{C or S} and payload syntax is either a plain action or a JSON string.
Return type:a MQTTMessage object
Raises:ValueError – in case a token conversion fails

6.11. mqttgateway.start_gateway module

Defines the function that starts the gateway.


Entry point.


Initialisation of the application and main loop.

Initialises the configuration and the log, starts the interface, starts the MQTT communication then starts the main loop. The loop can start in mono or multi threading mode. If the loop method is defined in the gateway_interface class, then the loop will operate in a single thread, and this function will actually loop forever, calling every time the loop method of the interface, as well as the loop method of the MQTT library. If the loop method is not defined in the gateway_interface class, then it is assumed that the loop_start method is defined and it will be launched in a separate thread. The priority given to the mono thread option is for backward compatibility.

The data files are:

  • the configuration file (compulsory), which is necessary at least to define the MQTT broker; a path to it can be provided as first argument of the command line, or the default path will be used;
  • the map file (optional), if the mapping option is enabled.

The rules for providing paths of files are available in the configuration file template as a comment. The same rules apply to the command line argument and to the paths provided in the configuration file.

Parameters:gateway_interface (class) – the interface class (not an instance of it!)

6.12. mqttgateway.throttled_exception module

An exception class that throttles events in case an error is triggered too often.

exception mqttgateway.throttled_exception.ThrottledException(msg=None, throttlelag=10, module_name=None)

Bases: exceptions.Exception

Exception class base to throttle events

This exception can be used as a base class instead of Exception. It adds a counter and a timer that allow to silence the error for a while if desired. Only after a given period a trigger is set to True to indicate that a number of errors have happened and it is time to report them.

It defines 2 members:

  • trigger is a boolean set to True after the requested lag;
  • report is a string giving some more information on top of the latest message.

The code using these exceptions can test the member trigger and decide to silence the error until it is True. At any point one can still decide to use these exceptions as normal ones, ignore the trigger and report members and just raise the exception as normal.


    #some statements that might raise your own exception derived from ThrottledException
except YourExceptionError as err:
    if err.trigger:
  • msg (string) – the error message, as for usual exceptions, optional
  • throttlelag (int) – the lag time in seconds while errors should be throttled, defaults to 10 seconds
  • module_name (string) – the calling module to give extra information, optional
_count = 0
_timer = 1655500847.113966