Add AI engine for spam (decidim#11319)

* Add events for comments

* Add events for debates

* Add events for meetings

* Update the proposals commands

* Refactor with_events

* Apply review recommendations

* Create decidim-ai module

Add Gitlab action workflow

Patch the generator

Running linters

Gemfiles

* change description

* Add language detection-service

Add language service
Normalize gems

* Add registry strategy (#253)

* Add SpamDetectionService class (#255)

* Add BayesStrategy (#256)

* Add BayesStrategy

* Add Bayes Analyzer

* Refactor strategy intialization process

* Change the pipeline working dir

* Fixing spam suite

* Revert event changes

* Revert event changes

* Add BayesStrategy (#256) (#257)

* Add BayesStrategy

* Add Bayes Analyzer

* Refactor strategy intialization process

* Running linters

* Add score calculation (#262)

* Add event handlers and spec data (#263)

* Add event handlers and spec data

* Fixing failng specs

* Fix Catgeory error in untrain

* fix decidim ai tests

* Refactor AI namespaces (#269)

* Add Strategy module

* Add more namespaces

* Add resources to be indexed (#254)

* Add resources to be analyzed

* mend

* Add action to reset train model (#270)

* Add documentation page (#275)

* Add Initiatives to antispam (#271)

* Redesign: rename tests (decidim#11184)

* Remove skip_unless_redesing_enabled calls in tests

* Remove redesign_helpers for tests

* Unskip and adapt REDESIGN_PENDING examples

* Remove deprecated examples

* Allow to configure the default header level of attributes titles in diff cell

* Ensure accessibility of h in collaborative draft show

* Recover accessibility tests in proposals

* Remove comments

* Unskip and adapt test

* Fix expected text of link in test

* Remove check for removed follow actions in participatory text index

* Unskip tests

* Unskip and adapt to vote proposal tests

* Remove deprecated methods

* Remove deprecations in tests

* Remove collaborative drafts and proposals deprecated m cells

* Adapt proposal m cell cache hash to l cell

* Adapt proposal m-cell specs to l-cell

* Replace use of proposal_m card

* Remove condition on redesign from test

* Rename shared examples for attachments to not use redesign term

* Remove tests for CTA behavior deprecated in processes card-g

* Avoid duplicated ids of highlighted participatory process in menu breadcrumb

* Recover skipped examples

* Recover accessibility tests in participatory processes

* Remove participatory process description specs and include expected contents with metadata content block enabled

* Unskip REDESIGN_PENDING tests

* Move common statistics cells tests to shared examples

* Remove unused process metadata item

* Remove deprecated proceses m card and add tests for l and g cards

* Remove deprecated process group m card cell

* Recover tests after redesign in initiatives

* Remove skipped deprecated tests

* Remove assemblies description page test and move examples to show page with appropriate content blocks

* Remove assembly m card and related tests

* Unskip tests after completing redesing

* Adapt sorting test to redesigned interface

* Remove deprecated sorting tests because the redesigned interface prevents incomplete sortings

* Unskip tests

* Remove post m card cell

* Remove reference to redesign from test

* Remove comment m card

* Unskip accessibility test in debates versions

* Remove deprecated test

* Remove deprecated comments about redesign

* Remove deprecated test

The proposals card-l does not show the supports count in the metadata,
so there is no difference when supports are enabled on proposals
component or not

* Remove useless reference to redesign in test

* Remove REDESIGN_PENDING comment: do not use card-l cell for media links

* Remove REDESIGN_PENDING comments

* Remove unused partial

* Unskip tests

* Remove conference m card cell

* Adapt authorizations tests to redesign

* Remove deprecated tests

* Avoid errors in upload modal when file validation fails

* Unskip tests

* Remove deprecated test

* Remove REDESIGN_PENDING comments

* Update map styles in proposals

* Uncomment tests

* Uncomment and adapt selector in tests

* Remove reference to redesign

* Fix linter offense

* Remove obsolete brakeman skip and update other in proposals

* Remove unused translations and add some wizards keys to ignore unused

* Update expected url in activity cell with comments activities

* Increase header levels in version cell to include h1

* Recover missing data attribute in hero content block

* Fix coordinates update in add_proposal js

* Hide map only if there is no address with coordinates

* Empty address field before changing it in test

* Add issue reference to skipped tests

* Display replies on deleted comments and keep then with AJAX deletion refresh

* Unskip test

* Update decidim-core/app/cells/decidim/version/show.erb

Co-authored-by: Alexandru Emil Lupu <[email protected]>

* Use h2 tag

* Move shared examples require to core

* Replace actions with method doing the same

* Fix test

* Remove references to redesign in shared examples

---------

Co-authored-by: Alexandru Emil Lupu <[email protected]>

* Redesign: refactor the scope of the main Javascript initializer (decidim#11317)

* set the scope of all functions that are initialized at startup

* scope the datepicker as well

* remove precated js stuff (testing)

* fix lint & place accordingly the initializations

* Redesign: deprecate foundation-datepicker from public site (decidim#11420)

* deprecate foundation-datepicker public site

* fix core tests

* fix datepicker tests

* fix lint and rubocop

* replace now with current

* fix tests

* fix more tests

* fix tests

* remove byebugs

* Redesign: don't submit forms on modal's cancel

* Update documentation for here map configuration (decidim#11447)

* Two-columns layout fix (decidim#11400)

* Column conversations fix

* suggested changes added to &__aside

* &__main updated

* Changes made to span on &__main

* test

* spacing cli test

* Disabler for pointer events (decidim#11449)

* Added a disabler for pointer event

* Improved workaround for the tabs +1

* Indentation line 19

* Fix test component (decidim#11436)

* Fix label position on report button (decidim#11435)

* Fix pipeline after decidim#11449 (decidim#11455)

* Upgrade postgres from 11 to 14 (decidim#11453)

* Fix initiatives ransack filters (decidim#11329)

* Revert "Lock ChromeDriver to the latest working version (decidim#11391)" (decidim#11458)

This reverts commit 2450714.

* Force capital starting letter for PR titles (decidim#11457)

* Force capital starting letter for PR titles

* Force the starting character to be capitalized

* Allow disabling password expiration (decidim#11135)

* Allow disabling password expiration

NIST does not recommend expiring passwords:
https://pages.nist.gov/800-63-FAQ/#q-b05

* Add documentation about the new option

* Fix linter

* Fix linter

* Fix autocomplete result list duplicates (decidim#11442)

* Fix autocomplete user group duplication / Add system test

* Fix test

* Redesign: organization colors (decidim#11451)

* refactor the organization colors

* typo

* remove specs about them_color input

* fix specs

* use old red for testing

* set the default redesign colors to the organization factory (color-contrast)

* replace the uses of legacy primary color

* Redesign: axe accessibility issues (decidim#11403)

* dropdown card does not require an id

* add aria-label to progress bar

* disable dropdown for desktop navigation

* Refactor the use of filter_text_for to be in views instead of helpers

* Allow passing an id to filter_text_for method

* remove element_id (testing)

* pass the id to the label

* distinguish filters and order controls

* distinguish all uses of dropdown-menus

* rename control

* don't use a list when you need divs

* disable dropdown for desktop navigation

* update code for emoji

* remove ids from tests

* fix core tests

* fix election tests

* fix tests

* remove profile__user-role class

* remove needless aria-attribute

* remove duplicated ids

* empty merge

* Include alternative title for images when there's nothing but the title

* disable dropdown for desktop

* add alt text to default image

* user profile must not be an h5

* duplicate_id and copy button SR text

* fix odd layout for edit initiative

* redesign committee requests form

* fix a11y issue on budgets project list

* mark layout item header al role=complementary since it's outside

* do not expand elements if the page has very few contents

* fix a11y issue on debates, sortitions and forms

* fix rubocop

* fix erblint

* use the units as the aria-label

* standarize arias for progress bars

* initiatives style glitch

---------

Co-authored-by: Eduardo Martinez Echevarria <[email protected]>

* Integrate Bullet in decidim-dev (decidim#11416)

* Initial commit on bullet integration

* remove uneeded condition

* Disable failing actions

* Disable failing workflows

* Disable all the workflows

* Refactor actions

* Revert generator change

* Redesign: comment button (decidim#11461)

* render the comment button depending of the settings

* enhance condition: initiatives is not a component

* handle comments_blocked? in case of null

* fix offenses

* refactor duplicated views (decidim#11459)

* Redesign: remove the item itself from nav menu (decidim#11452)

* remove the item itself from nav menu

* fix i18n

* fix specs

* fix specs

* Refactor Decidim jobs to inherit from Decidim::ApplicationJob (decidim#11468)

* Add Initiatives to antispam

* Add initiatives specs

---------

Co-authored-by: Eduardo Martínez <[email protected]>
Co-authored-by: Hugoren Martinako <[email protected]>
Co-authored-by: Sina Eftekhar <[email protected]>
Co-authored-by: Tom <[email protected]>
Co-authored-by: Antti Hukkanen <[email protected]>
Co-authored-by: Carlo Beltrame <[email protected]>
Co-authored-by: JoonasAapro <[email protected]>

* Fix rubocop

* Running linters

* FIx failing spec

* Gem revert

* Fix gem dependencies

* Fix test suite

* Fix spec

* Revert wicked_pdf version change

* Add spam csv dictionaries to the list of exceptions

* Additional CSV

* Spell checks

* Spell checks

* Add Autolabeler config for Ai module

* Fix specs

* Refactor Language Detection

* Refactor models

* Refactor reporting_user_email setting

* Add modules for jobs

* Refactor reporting user

* Refactor trained models

* Add user_classifier refactor

* Rename LanguageDetection ruby module

* Addded the text cleanup class

* Add some documentation

* Add ngrams to Spelling checks

* Fix rubocop

* Fix Gemfiles

* Update decidim-ai/README.md

Co-authored-by: Antti Hukkanen <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andrés Pereira de Lucena <[email protected]>

* Fix more issues from code review

* Add module documentation page structure

* Add initializer template

* Refactor language engine

* Add checks for classifier contract

* Fix pipeline

* Fix errors

* Fix the checkspel

* Fix the checkspeller

* Move the backed to Redis as default

* More fixes

* Apply suggestions from code review

Co-authored-by: Andrés Pereira de Lucena <[email protected]>

* Add doc links

* Fix core spec

* Fix the AI classifier

* Fix the specs

* Add safe navigation

* Apply suggestions from code review

Co-authored-by: Andrés Pereira de Lucena <[email protected]>

* Fix namespace

* Update install

* Add fallback connection

* Lints

* Update decidim-ai/README.md

Co-authored-by: Andrés Pereira de Lucena <[email protected]>

* Cleanup

* ENV vars

* More ENV

* Apply suggestions from code review

Co-authored-by: Andrés Pereira de Lucena <[email protected]>

* Remove host param from url uploader method on Conferences (decidim#13508)

* Lock Ubuntu to 22.04 in github actions (decidim#13513)

* fix translation bar css (decidim#13512)

* Add Linked Meetings feature to allow linking meetings to other spaces (decidim#13169)

* Add linked meetings feature to allow linking meetings to other participatory spaces (MeetingLink model)

* Update meeting components selctor to not show the current_component and show non published components

* Fix rubocop offenses

* Fix eslint offenses

* Fix erblint offenses

* Fix spelling linter

* Normalise locales

* Fix several admin/meetings specs

* Fix specs

* Add specs

* Fix rubocop offenses

* Disable assign button with private non transparent meeting

* Add tooltip to unlink action button

* Update meeting redirection message

* Fix linking a meeting to private and transparent spaces

* Fix unlink tooltip in dynamic elements

* Normalise locales

* Fix redirect callout, avoid using hash syntax

* Move redirect notice to the frontend

* Fix meeting cell when shown outside of a component

* Escape space names

* Fix meeting link in accountability results

* Fix meeting cell when url extra params are passed

* Unify private_non_transparent logic

* Fix rubocop offense

* Fix method

* Add specs for private_non_transparent_space? method in component

* Create a content block for open debates on the home of a space (decidim#13510)

* add content blocks for debates

* fix debetes order

* add specs

* change HighlightedDebatesCell

* change HighlightedDebatesCell

* refactor HighlightedDebatesCell

* Update decidim-debates/spec/cells/decidim/debates/content_blocks/highlighted_debates_cell_spec.rb

Co-authored-by: Alexandru Emil Lupu <[email protected]>

* fix test

---------

Co-authored-by: Alexandru Emil Lupu <[email protected]>

* Remove CLD

* Add User group tests

* More review

* more review

* Remove module dataset training

* Update docs/modules/develop/pages/ai_tools.adoc

Co-authored-by: Andrés Pereira de Lucena <[email protected]>

* Remove doc bits

* Apply suggestions from code review

---------

Co-authored-by: Eduardo Martínez <[email protected]>
Co-authored-by: Hugoren Martinako <[email protected]>
Co-authored-by: Sina Eftekhar <[email protected]>
Co-authored-by: Tom <[email protected]>
Co-authored-by: Antti Hukkanen <[email protected]>
Co-authored-by: Carlo Beltrame <[email protected]>
Co-authored-by: JoonasAapro <[email protected]>
Co-authored-by: Andrés Pereira de Lucena <[email protected]>
Co-authored-by: Ivan Vergés <[email protected]>
Co-authored-by: Màxim Colls <[email protected]>
Co-authored-by: Anna Topalidi <[email protected]>

.github/actions/spelling/excludes.txt

@@ -108,4 +108,8 @@
 ^\Qdecidim-meetings/app/views/decidim/meetings/admin/agenda/show.html.erb\E$
 ^\Qdecidim-participatory_processes/app/views/decidim/participatory_processes/participatory_processes/_statistics.html.erb\E$
 ^\Qdecidim-proposals/app/views/decidim/proposals/proposals/_endorsements_card_row.html.erb\E$
+^\Qdecidim-ai/data/blocked_accounts.csv\E$
+^\Qdecidim-ai/data/sms-spam.csv\E$
+^\Qdecidim-ai/data/spam_comments.csv\E$
+^\Qdecidim-ai/spec/support/test.csv\E$
 ignore$

.github/actions/spelling/expect.txt

@@ -5,6 +5,7 @@ abcd
 abcert
 abcloktpes
 accesslist
+Aceasta
 acentos
 acro
 activejob
@@ -21,9 +22,12 @@ admins
 adoc
 AFFERO
 AGPL
+aitools
 Ajuntament
 Ajuntamentde
 alabs
+alecslupu
+Alexandru
 amd
 amendables
 AMR
@@ -66,6 +70,7 @@ backticks
 barcat
 Basecontroller
 Batchloader
+bayes
 bbb
 Bbbb
 BCN
@@ -124,6 +129,7 @@ childs
 chromedriver
 cht
 cityhall
+cld
 clickable
 clickbait
 cmax
@@ -283,6 +289,7 @@ ERVz
 Esa
 espa
 estat
+este
 Etags
 etherpad
 evanfuture
@@ -503,6 +510,7 @@ lsclusters
 lte
 lteq
 LTREE
+Lupu
 lvh
 lvml
 magick
@@ -582,6 +590,7 @@ NEWCODE
 newsletterable
 newusermanager
 newwindow
+ngrams
 nicknamizable
 nicknamize
 nie
@@ -719,6 +728,7 @@ promotal
 promotors
 proposta
 Propostes
+propozitie
 Propuesta
 prosemirror
 protonmail
@@ -955,6 +965,7 @@ unresolveable
 unsigning
 unsigns
 unsubscribes
+untrains
 unvotes
 upcomingfoobar
 UPDREF

.github/autolabeler.yml

@@ -13,6 +13,7 @@ dependencies:
 javascript: "*.js"
 "module: accountability": decidim-accountability/**/*
 "module: admin": decidim-admin/**/*
+"module: ai": decidim-ai/**/*
 "module: api": decidim-api/**/*
 "module: assemblies": decidim-assemblies/**/*
 "module: blogs": decidim-blogs/**/*

.github/workflows/build_app.yml

@@ -21,6 +21,8 @@ jobs:
       DATABASE_USERNAME: postgres
       DATABASE_PASSWORD: postgres
       DATABASE_HOST: localhost
+      DECIDIM_SPAM_DETECTION_BACKEND_RESOURCE: "memory"
+      DECIDIM_SPAM_DETECTION_BACKEND_USER: "memory"
     services:
       postgres:
         image: postgres:14

.github/workflows/ci_ai.yml

@@ -0,0 +1,34 @@
+name: "[CI] Ai"
+on:
+  push:
+    branches:
+      - develop
+      - release/*
+      - "*-stable"
+  pull_request:
+    branches-ignore:
+      - "chore/l10n*"
+    paths:
+      - "*"
+      - ".github/**"
+      - "decidim-ai/**"
+      - "decidim-core/**"
+      - "decidim-dev/**"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  build_app:
+    uses: ./.github/workflows/build_app.yml
+    secrets: inherit
+    name: Build test application
+  main:
+    needs: build_app
+    name: Tests
+    uses: ./.github/workflows/test_app.yml
+    secrets: inherit
+    with:
+      working-directory: "decidim-ai"
+      test_command: bundle exec parallel_test --type rspec --pattern spec/

.github/workflows/ci_generators.yml

@@ -70,6 +70,8 @@ jobs:
       DATABASE_USERNAME: postgres
       DATABASE_PASSWORD: postgres
       DATABASE_HOST: localhost
+      DECIDIM_SPAM_DETECTION_BACKEND_RESOURCE: "memory"
+      DECIDIM_SPAM_DETECTION_BACKEND_USER: "memory"
     steps:
       - uses: actions/checkout@v4
         with:

.github/workflows/ci_performance_metrics_monitoring.yml

@@ -24,6 +24,8 @@ env:
   RAILS_ENV: development
   RAILS_BOOST_PERFORMANCE: "true"
   SHAKAPACKER_RUNTIME_COMPILE: "false"
+  DECIDIM_SPAM_DETECTION_BACKEND_RESOURCE: "memory"
+  DECIDIM_SPAM_DETECTION_BACKEND_USER: "memory"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
@@ -50,6 +52,8 @@ jobs:
       DATABASE_USERNAME: postgres
       DATABASE_PASSWORD: postgres
       DATABASE_HOST: localhost
+      DECIDIM_SPAM_DETECTION_BACKEND_RESOURCE: "memory"
+      DECIDIM_SPAM_DETECTION_BACKEND_USER: "memory"
     steps:
       - uses: actions/checkout@v4
         with:

.github/workflows/ci_production_check.yml

@@ -44,6 +44,8 @@ jobs:
       DATABASE_USERNAME: postgres
       DATABASE_PASSWORD: postgres
       DATABASE_HOST: localhost
+      DECIDIM_SPAM_DETECTION_BACKEND_RESOURCE: "memory"
+      DECIDIM_SPAM_DETECTION_BACKEND_USER: "memory"
     steps:
       - uses: actions/checkout@v4
         with:

.github/workflows/test_app.yml

@@ -54,6 +54,8 @@ jobs:
       DATABASE_USERNAME: postgres
       DATABASE_PASSWORD: postgres
       DATABASE_HOST: localhost
+      DECIDIM_SPAM_DETECTION_BACKEND_RESOURCE: "memory"
+      DECIDIM_SPAM_DETECTION_BACKEND_USER: "memory"
     services:
       validator:
         image: ghcr.io/validator/validator:latest

.spelling.yml

@@ -5,6 +5,8 @@ exclude_paths:
   - decidim-core/lib/decidim/db/common-passwords.txt
   - decidim-initiatives/spec/types/initiative_type_spec.rb
   - decidim-proposals/app/packs/documents/decidim/proposals/participatory_texts/participatory_text.md
+  - decidim-ai/data/(.*).csv
+  - decidim-ai/spec/support/test.csv
 
 forbidden:
   arent: are not

Gemfile

@@ -5,6 +5,7 @@ source "https://rubygems.org"
 ruby RUBY_VERSION
 
 gem "decidim", path: "."
+gem "decidim-ai", path: "."
 gem "decidim-conferences", path: "."
 gem "decidim-design", path: "."
 gem "decidim-initiatives", path: "."