Module EMail Collector

EMailCollector
Numero/ID of module	50320
User doc. of module	This page
Developer doc. of module	Module EMail Collector (developer)

Introduction

This module permits to collect emails from a mailbox using the IMAP protocol and record event or data in your Dolbarr using information found into email.

Installation

This module is included with the Dolibarr distribution, so there is no need to install it.

Setup

To use this module, you must first enable it using an administrator account, via the menu option "Home - Setup - Modules".

Choose the tab where the module is listed. Then click on "Activate".

The module is now activated.

If a cog icon appears on module thumb or at end of the line of the module, click on it to access the setup page specific to the module.

If available, it is recommended to enable the option "Use the PHP-IMAP library for IMAP instead of native PHP IMAP. This also allows the use of an OAuth2 connection for IMAP (module OAuth must also be activated)." so when creating an email collector, you will be able to choose between IMAP login with user/pass or with an OAuth2 token.

Create an email collector

Create as many email collectors you need.

Host of email IMAP server: Enter the full name of the IMAP server. Exemple: imap.gmail.com

Port of email IMAP server / Imap encryption: Choose 993 / SSL

Host charset: Choose UTF-8 (works 95% of time). May be US-ASCII with some version/setup of Office365.

Authentication mode

If you choose "Password", you must set an IMAP login and password.
If you choose "OAuth2", you must set the IMAP login and set the OAuth2 entry to use into the field Oauth service. To create your OAuth2 entry and choose the correct permission for this entry see documentation of Module OAuth

Source directory

To collect received email, enter here "INBOX"
To collect email sent, enter here "Sent"

Target directory A good practice is to move the collected email into another folder to avoid to have them collected twice. You must create the output target IMAP directory manually before running your email collector.

For example, with Gmail, to create a new IMAP directory:

Sign in to your Gmail account.
At the top right of the Gmail window, click Settings See all settings.
Click Labels.
Create Label and check the box "Show in IMAP"

Filters

Choose criteria to define which email is qualified for the collect.

Operations to do

Choose the action to do on collected emails.

Usage

Let the job scheduler (Module Scheduled jobs) being executed automatically to run the Email collectore and see your data being recorded into Dolibarr.

To avoid to wait the batch, you can force to execute the email collector manually from the Email collector record with the button "Run now". When making test, a good practice is to set the maximum number of email collected to a low value.

A test mode is available (button Test). This mode does not cause any modification of your mailbox (just reading) nor modification in Dolibarr. On the other hand, it allows you to see the different stages of the selection of qualified email, the filter and the processing of the actions that would be applied if we were not in test mode. This is an important feature to fine-tune the configuration of the collector.

Trouble shooting

If you got the error "connection setup failed - authenticate" with Google OAuth and your OAuth token + refresh works, check the Login (email) in your Emailcollector setup.

Example of use case: Using Emailcollector for a support center

You have a support center with n employees. The email of your support center is support@mycompany.com (an independant email or a mailing-list that dispatch any input email to the n employees).

You must:

Declare the email support@mycompany.com as a possible sender email for each employee (Example on Gmail: click on the menu wheel "Parameters", See All parameters, tab "Accounts and import", in section "Send email as", click on "Add another email", enter the email support@mycompany.com but uncheck the checkbox "alias", then click on link received by email to validate the new sender profile, go back in section "Send email as" and check the option "Reply from the same address the message was sent to").

So it will be possible for every employee to send/answer emails as the sender support@mycompany.com. So if the customer answer email of any employee, the answer will be sent to support@mycompany.com (so the n employees will also receive it).

Once this is done, if the email support@mycompany.com is a Mailing-List, it is recommended to create another email account, for exemple *followsupport@mycompany.com* and include it as another member of the mailing list support@mycompany.com. So followsupport@mycompany.com will receive any emails and answers inside the Inbox.
Enable the Module EMail Collector and the Module Ticket.
Create one EmailCollector entry with credentials (IMAP login or OAuth) of the *followsupport* user and Inbox as the input directory. You can set the output directoy as "Processed emails" (you must create the directory or label before the first run). As the inbox is dedicated to followsupport@mycompany.com email, no need to set filters.

Imagine that the received emails are formated by a web form and has this content:

Organization: A customer or name
Email: acustomeremail@example.com

Set the action to "Load the thirdparty by searching name/alias/email/id (load only)" with parameters

name=EXTRACT:BODY:Organization:\s*([\w\-\s]+)$
email=EXTRACT:BODY:Email:\s*([\w\.\-]+@[a-zA-Z\.\-]+)

If the received email has no known format, you can't reuse content to guess the company name. So only the sender email will be used.

Then set a second action that is "Create or complete a ticket" (no parameters).

So each time an email is received to support@mycompany.com, or an answer is done, the user follow@mycompany.com receive the email and the collector will use it, trying to guess if it is a input or output email and will record a help desk ticket or complete a ticket..

Example of use case: Create an opportunity for a team of sales

You have a sale team with n employees. The email of your team is sales@mycompany.com (an independant email or a mailing-list that dispatched any input email to the n employees).

You must:

Declare the email sales@mycompany.com as a possible sender email for each employee (but uncheck the checkbox "alias" if using Gmail). So it will be possible for every employee to send/answer emails as the sender sales@mycompany.com. So if the customer answer email of any employee, the answer will be sent to sales@mycompany.com (so the n employee will also receive it).

Once this is done, if the email sales@mycompany.com is a Mailing-List, it is recommended to create another email account, for exemple *followsales@mycompany.com* and include it as another member of the mailing list sales@mycompany.com. So followsales@mycompany.com will receive any emails and answers inside its Inbox.
Enable the Module EMail Collector and the Module Projects with option "Allow opportunity management" to On.
Create one EmailCollector entry with credentials (IMAP login or OAuth) of the sales mailbox. You can set the output directoy as "Processed emails" (you must create the directory or label before the first run). As the Inbox is dedicated to followsales@mycompany.com email, no need to set filters.

Imagine that the received emails are formated by a web form and has this content:

Organization: A customer or name
Email: acustomeremail@example.com

Set the action to "Load the thirdparty by searching name/alias/email/id (create if not found)" with parameters

name=EXTRACT:BODY:Organization:\s*([\w\-\s]+)$
email=EXTRACT:BODY:Email:\s*([\w\.\-]+@[a-zA-Z\.\-]+)

If the received email has no know format, you can't reuse content of email to guess the company name. So only the sender email will be used.

Then set a second action that is "Create an opportunity" (no parameters).

So each time an email is received to sales@mycompany.com, or an answer is done, the user followsales@mycompany.com receive the email and the collector will use it to find existing thirdparty or will create it if not found, then will create the Project/opportunity attached to it.

Boundary Value Analysis

Use Cases

The following use cases were tested for some version of Dolibarr and mail providers:

Use Case	GMail + Dolibarr 20	Errors / Problems
Collect received mails to create a ticket (created from scratch by someone)	OK
Answer the ticket email received by email, this answer appears into the "Email sent", the collector collect the email sent and link it to the ticket.	OK
Send email from a Dolibarr ticket. The email must appears into "Email Sent" of the mailer, but the email collector must not collect it from the "Email sent" because it will be a double: When the email is generated and sent from Dolibarr (so saved into Dolibarr), and the SMTP platform also save the email into the "Emails sent", the collector we must not re-import it.	???
Collect and link email to the ticket for email received from the customer answer of our answer (sent from ticket interface).	OK
Collect and link email to the ticket for email received from the customer answer of our answer (sent from the emailer)	OK
Collect and link email to the ticket for email sent from the emailer on a mail that was an answer of a ticket.	OK

Filters

Depending on your mailbox service, some filters may works or not. This is a summary of tests done with different email services. When your mail hoster does not support the filter, Dolibarr may use the PHP to have filter working despite of this. But this is not always implemented.


Filters/Technology used to filter	GMail	Dolibarr+PHP	Errors / Problems
From	OK	NA
To	OK	NA
Cc	KO	?	Search string not understood - Unknown search criterion: CC
Bcc	KO	?	Search string not understood - Unknown search criterion: BCC
ReplyTo	NA	Filter support with Dolibarr 20+
Subject	OK	NA
Body	OK	NA

Already read	OK	NA
Unread	OK	NA
Unanswered	OK	NA
Answered	OK	NA
Size (Smaller than)	KO	NA	Search string not understood - Unknown search criterion: SMALLER
Size (Larger than)	KO	NA	Search string not understood - Unknown search criterion: LARGER
Message from a conversation initiated by a first email sent from Dolibarr	NA	OK
Message from a conversation initiated by a first email NOT sent from Dolibarr	NA	OK
Message sent from Dolibarr	NA	OK
Message NOT sent from Dolibarr	NA	OK	Message not processed but not an error

Is not answer (initial email)	NA	OK
Is an answer of an initial email	NA	OK