Dolibarr ERP CRM Wiki

Changes

User:Jtraulle/DiscourseMigration (view source)

Revision as of 15:54, 24 May 2023

6,936 bytes added ,  1 year ago
m
→‎Allow PostgreSQL to be accessed from an external application
Line 1: Line 1:  +
{{DISPLAYTITLE:Discourse Migration}}
 +
 
This page gathers info about Discourse migration.
 
This page gathers info about Discourse migration.
 +
 +
For the implementation, see [[User:Jtraulle/DiscourseMigration/Implementation]]
 +
 +
 +
==Context (forum migration project genesis)==
 +
 +
 +
Topic has been first discussed on the french speaking Dolibarr forum here : https://www.dolibarr.fr/forum/11-suggestionsnouvelles-fonctionnalites/64778-forum-quelle-solution-choisir
 +
 +
Topic has also been discussed on [[Report board meeting 20190730 teleconf]]
      Line 81: Line 93:  
This explains why there is no '''Résolu''' tag in the proposal.
 
This explains why there is no '''Résolu''' tag in the proposal.
 
<br />
 
<br />
−
<br /><br />
+
<br />
 +
 
 +
===Proposed mapping between current and new skeleton===
 +
 
 +
 
 +
Categories are in bold and sub-categories in italic.
 
{| class="wikitable mw-collapsible"
 
{| class="wikitable mw-collapsible"
−
|+Proposed mapping between current and new skeleton
+
|+
−
''<small>Categories are in bold and sub-categories in italic</small>''
   
!Current categories or sub-categories
 
!Current categories or sub-categories
 
!New proposed categories and sub-categories
 
!New proposed categories and sub-categories
Line 319: Line 335:  
|''Projets/Opportunités/Affaires''
 
|''Projets/Opportunités/Affaires''
 
|}
 
|}
−
<br />
+
 
 +
 
 +
===Preview of new mapping after migration===
 +
<br /><gallery widths="400" heights="800">
 +
File:Categories-discourse.png|Preview of new mapping after migration
 +
</gallery><gallery widths="500" heights="300">
 +
File:Liste-cat.png|Posts distribution across main categories
 +
File:Liste-sub-cat.png|Posts distribution across sub-category "Utiliser mon Dolibarr"
 +
</gallery>
 +
 
 +
 
 +
==Demo instance==
 +
 
 +
Demo instance with migrated data is available here : https://test-dolibarr-discourse.traulle.net/ (private ; see https://www.dolibarr.fr/forum/administration-forum/64852-instance-de-test-discourse#113680 for access - restricted to forum moderators and admins)
 +
 
    
==Migration path==
 
==Migration path==
Line 338: Line 368:     
===Import Users===
 
===Import Users===
−
* Retrieve old "anonymous" Kunena users that does not have a Joomla! account
  −
* Retrieve profile info
  −
* Retrieve avatars
     −
=== Import Categories===
+
*Retrieve old "anonymous" Kunena users that does not have a Joomla! account
−
* Import categories hierarchy ignoring main groups (Discourse only has two levels hierarchy)
+
*Retrieve profile info
−
* Categories are imported preserving the original order
+
*Retrieve avatars
−
* Categories with no posts (and no subcategories) are ignored/skipped  
+
 
 +
===Import Categories===
 +
 
 +
*Import categories hierarchy ignoring main groups (Discourse only has two levels hierarchy)
 +
*Categories are imported preserving the original order
 +
*Categories with no posts (and no subcategories) are ignored/skipped
 +
 
 +
------
 +
 
 +
The latest version of the custom migration script takes care of re-categorizing topics into the [[#Proposed_skeleton]]
    
===Import Posts===
 
===Import Posts===
−
* Multiple enhancements on posts text content
  −
** Escape some commonly used chars (>, -, +, *) to not be wrongly parsed as Markdown
  −
** Block quote
  −
*** add a line break after [quote] BBCode tag if there is none
  −
*** Replacing old posts reference in quote blocks by new ones
  −
*** Removing remaining posts references in quote blocks for posts that does not exists anymore
  −
** Unordered lists
  −
*** Break line after [ul] only for lines that starts by [ul] and followed by [li] tag on the same line
  −
** Strike support
  −
*** Replace [strike][/strike] BBCode tag by <s></s> HTML equivalent for markdown parser
  −
** Emojis
  −
*** Convert some emojis shortcuts to preserve custom emojis previously used
     −
* Thank you
+
*Multiple enhancements on posts text content
−
** Kunena post's "thank you" are imported as Discourse post's likes  (added)
+
**Escape some commonly used chars (>, -, +, *) to not be wrongly parsed as Markdown
 +
**Block quote
 +
***add a line break after [quote] BBCode tag if there is none
 +
***Replacing old posts reference in quote blocks by new ones
 +
***Removing remaining posts references in quote blocks for posts that does not exists anymore
 +
**Unordered lists
 +
***Break line after [ul] only for lines that starts by [ul] and followed by [li] tag on the same line
 +
**Strike support
 +
***Replace [strike][/strike] BBCode tag by <s></s> HTML equivalent for markdown parser
 +
**Emojis
 +
***Convert some emojis shortcuts to preserve custom emojis previously used
 +
 
 +
*Thank you
 +
**Kunena post's "thank you" are imported as Discourse post's likes  (added)
    
===Import Attachments===
 
===Import Attachments===
Line 371: Line 408:  
To download attachments from previous forum, start by export the URLs from the Kunena database.
 
To download attachments from previous forum, start by export the URLs from the Kunena database.
   −
<source lang="SQL">
+
<syntaxhighlight lang="SQL">
 
SELECT CONCAT('https://www.dolibarr.fr/', folder, '/', filename) AS url FROM gvrsi_kunena_attachments ORDER BY id;
 
SELECT CONCAT('https://www.dolibarr.fr/', folder, '/', filename) AS url FROM gvrsi_kunena_attachments ORDER BY id;
−
</source>
+
</syntaxhighlight>
    
Put all the URLs into a text file (for example <code>uploads.txt</code>).
 
Put all the URLs into a text file (for example <code>uploads.txt</code>).
Line 379: Line 416:  
Next, use <code>wget</code> to download all attachment preserving the directory structure (yuuup, file names of multiple distinct attachments can be the same ...)
 
Next, use <code>wget</code> to download all attachment preserving the directory structure (yuuup, file names of multiple distinct attachments can be the same ...)
   −
<source lang="bash">
+
<syntaxhighlight lang="bash">
 
wget --no-host-directories --force-directories --input-file=uploads.txt
 
wget --no-host-directories --force-directories --input-file=uploads.txt
−
</source>
+
</syntaxhighlight>
 +
 
 +
===Permalinks===
 +
 
 +
Permalinks offer the ability to preserve old forum links (from Kunena) and emit a HTTP 301 (Permanent Redirect) when hitting such old links for categories, topics, posts and profile links.
 +
 
 +
====Categories permalinks====
 +
 
 +
Kunena stores aliases for categories in the '''kunena_categories''' '''alias''' database field.
 +
 
 +
====Topics permalinks====
 +
 
 +
For topics URLs, this a bit more tricky since they are not stored in database (it uses standard Joomla! SEO friendly URLs generated using JRouter) and should therfore be generated on the fly in the import script.
 +
 
 +
Research showed that a "safe" alias can be generated using following Joomla! function :
 +
* <code>stringUrlSafe()</code> function in <code>libraries/vendor/joomla/filter/src/OutputFilter.php</code>
 +
* using <code>transliterate()</code> function in <code>libraries/src/Language/Language.php</code>
 +
* using <code>utf8_latin_to_ascii()</code> function in <code>libraries/src/Language/Transliterate.php</code>
 +
 
 +
====Direct messages permalinks====
 +
 
 +
Permalinks to specific messages like for example https://www.dolibarr.fr/forum/12-howto--aide/62681-wiki-dolibarr-org?start=225#113400 are even more tricky than topics permalinks because everything that is after a # in considered and anchor and, as such could be treated only on the client side (therefore using Javascript).
 +
 
 +
The permalink treatment logic for direct messages is the following :
 +
 
 +
# User go to https://test-dolibarr-discourse.traulle.net/forum/12-howto--aide/62681-wiki-dolibarr-org?start=225#113400 (the old permalink)
 +
# Normalization permalink rule <code>/(.*)\?.*/\1</code> is triggered and therefore remove part of the URL starting by ? ; the new URL to be processed become https://test-dolibarr-discourse.traulle.net/forum/12-howto--aide/62681-wiki-dolibarr-org#113400
 +
# Permalink for topic is triggered and redirect the URL to https://test-dolibarr-discourse.traulle.net/forum/t/wiki-dolibarr-org/27878#113400 (keeping the original anchor with the old postid)
 +
# Javascript detect that the current URL have a # part
 +
# Using Javascript, we redirect to the permalink https://test-dolibarr-discourse.traulle.net/forum/old-post/113400 that redirects to https://test-dolibarr-discourse.traulle.net/forum/t/wiki-dolibarr-org/27878/227
 +
 
 +
 
 +
=====Javascript part=====
 +
 
 +
This code is to paste in the <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Common → </head></code> section.
 +
 
 +
<syntaxhighlight lang="HTML">
 +
<script type="text/javascript">
 +
var urlSplit = document.URL.split("#");
 +
if (urlSplit[1]) {
 +
    location.href = "/forum/old-post/" + urlSplit[1];
 +
}
 +
</script>
 +
</syntaxhighlight>
 +
 
 +
===Ads===
 +
 
 +
Current forum display an Ad banner (ads are currently managed in house using Joomla!
 +
 
 +
Discourse offers an ''' Official Advertising / Ad Plugin''' that can be used as a replacement. See https://meta.discourse.org/t/official-advertising-ad-plugin-for-discourse/33734
 +
 
 +
 
    
==Tentative design==
 
==Tentative design==
−
<br />
+
 
    
===Custom logos===
 
===Custom logos===
Line 432: Line 520:     
====Custom Header to add====
 
====Custom Header to add====
−
Go to <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Common → Header</code> and paste :
+
Go to <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Desktop → Header</code> and paste :
 
<br />
 
<br />
 
{| class="mw-collapsible mw-collapsed"
 
{| class="mw-collapsible mw-collapsed"
Line 483: Line 571:  
<br />
 
<br />
 
====Custom Footer to add====
 
====Custom Footer to add====
−
Go to <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Common → Footer</code> and paste :
+
Go to <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Desktop → Footer</code> and paste :
    
<syntaxhighlight lang="HTML">
 
<syntaxhighlight lang="HTML">
Line 492: Line 580:  
====Custom CSS to add====
 
====Custom CSS to add====
   −
Go to <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Common → CSS</code> and paste :<br />
+
Go to <code>Admin → Customize → Themes → Light → Edit CSS/HTML → Desktop → CSS</code> and paste :<br />
 
{| class="mw-collapsible mw-collapsed"
 
{| class="mw-collapsible mw-collapsed"
 
!
 
!
Line 718: Line 806:  
|}
 
|}
 
<br />
 
<br />
−
      
==Requirements==
 
==Requirements==
Line 737: Line 824:  
Discourse '''<u>requires</u>''' a FQDN (Fully Qualified Domain Name) : for example <code>forum.dolibarr.fr</code> and cannot be installed in a subdirectory.
 
Discourse '''<u>requires</u>''' a FQDN (Fully Qualified Domain Name) : for example <code>forum.dolibarr.fr</code> and cannot be installed in a subdirectory.
    +
=> We must found a solution and valid it works.
 +
Future is to have doc/wiki on https://www.dolibarr.org/wiki, an forum of https://www.dolibarr.org/forum. Forum is 2/3 of traffic received on dolibarr.org and it is important to have a lot of content trusted by Google to have dolibarr.org valorized highly on Google search engine answers.
 +
 +
There is a solution here using a CDN frontend like Fastly: https://meta.discourse.org/t/discourse-in-a-subfolder-multiple-servers-sharing-a-domain/30514
 +
It is based on Fastly, be Dolibarr is using Cloudflare as CDN that may offers same features.
 +
 +
Another solution without CDN, using your own proxy, is suggested here https://meta.discourse.org/t/subfolder-support-with-docker/30507 but previous one seems easier.
 +
 +
 +
=> '''Yes, it is doable using nginx or Apache2 as a reverse proxy.''' In fact, I am using nginx as a reverse proxy on the demo instance.
 +
 +
=> With Cloudflare I found this : https://blog.cloudflare.com/subdomains-vs-subdirectories-improved-seo-part-2/
    
===Email requirements===
 
===Email requirements===
Line 765: Line 864:  
*Configure / allow / unblock port 587 for outgoing email sending by postfix in the firewall (IP table, etc.)
 
*Configure / allow / unblock port 587 for outgoing email sending by postfix in the firewall (IP table, etc.)
 
*Configure Postfix to require authentication : see https://blog.rom1v.com/2010/01/ajouter-lauthentification-smtp-sur-un-serveur-mail/
 
*Configure Postfix to require authentication : see https://blog.rom1v.com/2010/01/ajouter-lauthentification-smtp-sur-un-serveur-mail/
 +
 
------
 
------
 
Alternative : https://forum.normandie-libre.fr/t/installation-dun-forum-discourse-avec-apache-docker-et-mailboy/25
 
Alternative : https://forum.normandie-libre.fr/t/installation-dun-forum-discourse-avec-apache-docker-et-mailboy/25
Line 771: Line 871:  
*[https://fr.mailjet.com/ Mailjet] pricing, [https://fr.mailjet.com/pricing/ first 6 000 monthly emails free (no more than 200 email per day ; 30 000 per month for 7,16€])
 
*[https://fr.mailjet.com/ Mailjet] pricing, [https://fr.mailjet.com/pricing/ first 6 000 monthly emails free (no more than 200 email per day ; 30 000 per month for 7,16€])
 
*[https://aws.amazon.com/fr/ses/ AWS SES (Amazon Web Services Simple Email Service)] pricing, [https://aws.amazon.com/fr/ses/pricing/ 0.10$ per 1 000 emails].
 
*[https://aws.amazon.com/fr/ses/ AWS SES (Amazon Web Services Simple Email Service)] pricing, [https://aws.amazon.com/fr/ses/pricing/ 0.10$ per 1 000 emails].
 +
*[https://sendgrid.net/ SendGrid] pricing, Free for 100 emails per day or 14$ per month for 40 0000 emails.
 +
 
|}
 
|}
    
==Tips and tricks==
 
==Tips and tricks==
 +
 +
===Discourse tips and tricks===
 +
Add an Unreplied tab to the navbar : [https://meta.discourse.org/t/reply-reminder-remind-users-to-reply-to-new-users-topics-with-zero-replies/42644/2]
 +
 +
 +
====Execute migration script====
 +
 +
<syntaxhighlight lang="bash">
 +
/var/discourse/launcher enter web_only
 +
# Install required dependencies to import data (gems)
 +
su discourse -c "export IMPORT=1; bundle config unset deployment; bundle config set path 'vendor/bundle'; bundle config set without 'test development'; bundle install --jobs 4"
 +
# Run import script
 +
su discourse -c 'export IMPORT=1; cp /scripts/kunena3_dolibarr_de.rb /var/www/discourse/script/import_scripts/kunena3_dolibarr_de.rb; bundle exec ruby /var/www/discourse/script/import_scripts/kunena3_dolibarr_de.rb'
 +
</syntaxhighlight>
 +
 +
Note :
 +
 +
====Restaure Discourse backup from command line====
 +
 +
<syntaxhighlight lang="bash">
 +
/var/discourse/launcher enter web_only
 +
su discourse -c 'bundle exec ruby script/discourse restore forum-dolibarr-france-2019-08-07-080937-v20190731090219.tar'
 +
</syntaxhighlight>
 +
 +
You must of course replace <code>forum-dolibarr-france-2019-08-07-080937-v20190731090219.tar</code> by the backup filename to restore.
 +
 +
Note : to restore a save from a multisite install that is not the main (first) database, prepend the command by <code>RAILS_DB=dolibarrde</code> where <code>dolibarrde</code> is the name of the database :
 +
 +
<syntaxhighlight lang="bash">
 +
/var/discourse/launcher enter web_only
 +
su discourse -c 'RAILS_DB=dolibarrde bundle exec ruby script/discourse restore dolibarr-germany-forum-2020-08-16-163120-v20200814081437.sql.gz'
 +
</syntaxhighlight>
    
===Docker tips and tricks===
 
===Docker tips and tricks===
Line 785: Line 919:  
The default mount point for all persisted data related to Discourse is through  
 
The default mount point for all persisted data related to Discourse is through  
   −
<code>/var/discourse/shared/standalone</code>
+
<code>/var/discourse/shared/data</code> and <code>/var/discourse/shared/web-only</code>
−
 
      
===PostgreSQL tips and tricks===
 
===PostgreSQL tips and tricks===
Line 794: Line 927:  
''Useful to be able to access the database from a graphical client (like pgAdmin 4, TablePlus or Postico for example).''
 
''Useful to be able to access the database from a graphical client (like pgAdmin 4, TablePlus or Postico for example).''
   −
1. Expose the 5432 TCP port from the container to the host by adding this line to the '''<code>expose:</code>''' section of <code>containers/app.yml</code> :
+
1. Expose the 5432 TCP port from the container to the host by adding this line to the '''<code>expose:</code>''' section of <code>/var/discourse/containers/app.yml</code> :
 
<syntaxhighlight lang="yaml">
 
<syntaxhighlight lang="yaml">
 
  - "5432:5432"
 
  - "5432:5432"
Line 800: Line 933:  
2. Rebuild the container to take into account modifications to '''<code>app.yml</code>''' :
 
2. Rebuild the container to take into account modifications to '''<code>app.yml</code>''' :
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
−
  ./launcher rebuild app
+
cd /var/discourse
 +
  ./launcher rebuild data
 
</syntaxhighlight>
 
</syntaxhighlight>
 
3. Enter into the container, login as the default postgresql user, add a new user and grant it all perms to the database
 
3. Enter into the container, login as the default postgresql user, add a new user and grant it all perms to the database
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
−
  ./launcher enter app
+
cd /var/discourse
 +
  ./launcher enter data
 
  sudo -u postgres psql discourse
 
  sudo -u postgres psql discourse
 
</syntaxhighlight>
 
</syntaxhighlight>
Retrieved from "https://wiki.dolibarr.org/index.php?title=Special:MobileDiff/43983...56587"