feat(attachments, storage): u#3727 t#3961 physical attachment deletion system

[bameda]

This PR ensures that the files are deleted when the attachments are deleted. Several things must be taken into consideration:

• Currently, files are saved on disk, but this will change so we will have to use django's storage module.
• In the future we will have to postpone the deletion and make it easily recoverable (to avoid data loss). This is in line with the logical deletion of objects.

The solution that has been reached consists of:

• A new module taiga.commons.storage where everything related to file storage will be created.
• A new StoragedObject model that will store the information about where the file is stored.
• Now the Attachment model will have a ForeingKey to StoragedObject
• A signal is defined for Attachment during the post_save that will mark the associated StoragedObject as deleted.
• A periodic task is created that will physically delete all files that belong to StoragedObject records that have been deleted more than X days ago. (The frequency of the periodic task and time that the files will be kept are configurable parameters).
• There is a CLI command to remove deleted StoragedObject items and their files.

How to tests:

• Review the code.
• Tests are green.
• Manual tests: Tests logical deletion:
  • Init:
    • 👸 Open the backend admin panel and access to Storaged Objects list view
    • 👤 Open the frontend
  • Tests delete attachment:
    • 👤 Go to a story and upload some attachments
    • 👸 refresh and see new storaged objets created
    • 👤 Delete attachments
    • 👸 refresh and see storaged objets with their deleted datetime.
  • Tests delete project:
    • 👤 Create a new project.
    • 👤 Create a new story and upload some attachments (you can create more than one story with attachments)
    • 👸 refresh and see new storaged objets created
    • 👤 Delete the project
    • 👸 refresh and see all the storaged objets with their deleted datetime.
  • [Optional] Tests delete user with workspaces and projects with stories with atachments:
    • You can test a more complex scenario: deleting a user involves deleting workspaces that have projects with stories "full" of attachments.
• Manual tests: Tests physical deletion:
  • Test CLI: python -m taiga storage clean-storaged-objects --help
  • Test periodic task: You can modify TAIGA_STORAGE__DAYS_TO_STORE_DELETED_STORAGED_OBJECTS (int or None for all) and TAIGA_STORAGE__DAYS_TO_STORE_DELETED_STORAGED_OBJECTS (use cron format)

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/taiga
__main__.py	71	71	0%	17–155
wsgi.py	15	15	0%	8–31
src/taiga/attachments
admin.py	27	3	90%	25, 28, 70
events.py	8	2	75%	15, 20
models.py	19	2	89%	60, 63
repositories.py	49	2	93%	62, 64
src/taiga/auth/services
__init__.py	47	0	98%	90–>93
src/taiga/base
mocks.py	74	12	80%	93–94, 107–118, 143–>exit
src/taiga/base/api
permissions.py	49	1	99%	22
requests.py	11	3	80%	15, 21–23
src/taiga/base/db
__init__.py	4	1	75%	15
commands.py	54	54	0%	8–271
sequences.py	45	2	97%	91–92
src/taiga/base/db/admin
forms.py	14	8	38%	17–25
utils.py	11	7	36%	17–23
src/taiga/base/db/models
fields.py	15	1	88%	20
src/taiga/base/django
__init__.py	5	2	60%	17–19
settings.py	30	2	93%	16–17
urls.py	5	5	0%	8–13
wsgi.py	4	4	0%	8–14
src/taiga/base/i18n
__init__.py	102	9	88%	171–>170, 173–>170, 192–193, 203–205, 215, 268, 279, 283
choices.py	34	2	91%	80, 87
commands.py	74	74	0%	8–108
src/taiga/base/logging
formatters.py	24	10	50%	22–>exit, 38, 44–45, 51–60
middlewares.py	23	1	96%	20
src/taiga/base/repositories
neighbors.py	39	3	91%	43–44, 65
src/taiga/base/sampledata
commands.py	11	11	0%	8–40
constants.py	15	15	0%	14–51
demo_data.py	273	273	0%	8–529
factories.py	122	122	0%	9–353
test_data.py	290	290	0%	8–634
src/taiga/base/serializers
fields.py	20	1	96%	21
src/taiga/base/utils
commands.py	31	31	0%	8–60
datetime.py	44	18	60%	54, 67, 126–147
enum.py	14	1	92%	43
images.py	26	3	87%	21–22, 30
json.py	33	15	45%	30–35, 37, 39–44, 46, 50
pprint.py	6	6	0%	8–17
slug.py	23	4	85%	22–25
src/taiga/base/validators/fields
i18n.py	16	2	91%	19–20
uuid.py	15	2	90%	19–20
src/taiga/comments
admin.py	29	3	92%	16, 20, 24
events.py	11	3	73%	15, 20, 25
models.py	20	2	91%	43, 46
src/taiga/commons/storage
commands.py	13	13	0%	8–40
models.py	16	2	88%	38, 41
services.py	9	0	92%	17–>16
tasks.py	9	9	0%	8–19
src/taiga/conf
__init__.py	93	3	95%	109, 125–126
tokens.py	17	1	90%	34
src/taiga/emails
commands.py	35	35	0%	9–68
sender.py	18	5	71%	48–50, 54–73
tasks.py	39	4	90%	44–45, 53–54
src/taiga/events
app.py	18	1	92%	21
events.py	16	2	88%	31–32
logging.py	39	23	31%	18–>exit, 32–48, 51–60
manager.py	117	11	90%	78–79, 223–233, 239–242
responses.py	33	6	84%	33, 37–41
subscriber.py	80	4	95%	51, 55, 88, 91, 108–>120
src/taiga/events/actions
auth.py	23	11	44%	27–32, 39–45
event_handlers.py	9	4	56%	18–20, 28–30
projects.py	55	30	38%	36–40, 48–64, 72–80, 88–96
workspaces.py	55	30	38%	36–40, 48–64, 72–80, 88–96
src/taiga/events/pubsub/backends
base.py	30	1	98%	53
redis.py	66	5	94%	41–43, 85–86
src/taiga/exceptions/api
__init__.py	27	0	97%	28–>30
handlers.py	27	2	90%	27–28
middlewares.py	26	2	87%	37, 41
src/taiga/integrations/auth
services.py	39	7	82%	50–56
src/taiga/integrations/github
services.py	40	0	98%	63–>68
src/taiga/mediafiles
admin.py	20	2	91%	24, 27
models.py	26	2	92%	84, 87
src/taiga/permissions
services.py	103	1	98%	28, 106–>109
src/taiga/permissions/validators
fields.py	24	3	88%	18–19, 45
src/taiga/projects/invitations
models.py	25	2	92%	89, 92
repositories.py	84	1	98%	135
src/taiga/projects/invitations/api
validators.py	65	7	89%	38–40, 65–66, 84–85
src/taiga/projects/invitations/events
__init__.py	33	1	91%	37–>exit, 49–>exit, 74–>exit, 102
src/taiga/projects/invitations/services
__init__.py	183	1	99%	58–>53, 286
src/taiga/projects/memberships
models.py	16	2	88%	51, 54
repositories.py	86	2	97%	91, 124
src/taiga/projects/memberships/events
__init__.py	16	2	88%	18–24
src/taiga/projects/memberships/services
__init__.py	45	1	95%	64–>69, 111
src/taiga/projects/projects
admin.py	53	12	75%	33–34, 39–42, 50–51, 56–59, 90, 94
models.py	60	8	86%	64, 67, 78, 112, 115, 118–120
repositories.py	115	4	96%	125, 126–>129, 155, 254–255
tasks.py	6	1	88%	17
src/taiga/projects/projects/services
__init__.py	105	5	92%	77, 86, 192, 198–>201, 206, 228
src/taiga/projects/references
mixins.py	13	0	93%	25–>28
src/taiga/projects/roles
api.py	35	3	93%	46–48
models.py	26	3	86%	50, 53, 57
repositories.py	44	1	96%	92
src/taiga/projects/roles/services
__init__.py	25	1	94%	24, 57–>62
src/taiga/stories/assignments
models.py	15	2	87%	42, 45
repositories.py	52	1	97%	49
src/taiga/stories/attachments
api.py	42	1	96%	128
src/taiga/stories/comments
api.py	58	1	97%	177
src/taiga/stories/stories
admin.py	25	5	84%	76–81, 85, 89
models.py	25	2	92%	73, 76
src/taiga/stories/stories/services
__init__.py	113	2	97%	233, 256
src/taiga/tasksqueue
app.py	20	2	86%	29–30
commands.py	65	65	0%	8–122
logging.py	33	17	40%	18–>exit, 29–39, 42–51
manager.py	56	13	77%	37, 41, 45, 49, 94, 163, 177, 186, 191, 198, 202, 209, 213
task.py	12	1	92%	23
src/taiga/tokens
admin.py	44	10	81%	67, 70, 73, 76, 104, 108, 112, 116, 120, 124
models.py	30	4	87%	59, 62, 78, 81
src/taiga/users
admin.py	51	7	81%	32, 41, 50, 87–95
commands.py	9	9	0%	8–26
models.py	65	10	86%	71, 74, 90, 93, 97, 100, 103, 106, 127, 130
repositories.py	204	3	98%	209–>212, 335, 428–429
src/taiga/users/api
__init__.py	66	2	96%	216, 304
src/taiga/users/api/validators
__init__.py	42	3	92%	39–41
src/taiga/users/services
__init__.py	244	7	95%	75, 259, 262–>257, 297–>280, 306–>280, 322–>318, 345–>341, 366, 494–>499, 513–520, 521–>526, 524–525
src/taiga/workflows
models.py	36	4	89%	39, 42, 72, 75
src/taiga/workflows/services
__init__.py	91	3	94%	171, 198, 269–>277, 284
src/taiga/workspaces/invitations
models.py	24	2	92%	84, 87
permissions.py	9	1	82%	19
repositories.py	77	1	98%	123
src/taiga/workspaces/invitations/api
validators.py	18	2	88%	24–25
src/taiga/workspaces/invitations/events
__init__.py	25	1	91%	36–>exit, 60–>exit, 71
src/taiga/workspaces/invitations/services
__init__.py	123	1	99%	222
src/taiga/workspaces/memberships
api.py	43	1	96%	126
models.py	15	2	87%	43, 46
repositories.py	63	2	95%	72, 105
src/taiga/workspaces/memberships/services
__init__.py	46	1	97%	117
src/taiga/workspaces/workspaces
admin.py	57	19	62%	25–26, 31–34, 49, 52–57, 61, 65, 73–74, 79–82
models.py	18	2	90%	32, 35
repositories.py	127	2	99%	232–233
src/taiga/workspaces/workspaces/services
__init__.py	54	2	94%	113, 116
TOTAL	9779	1587	82%

Tests	Skipped	Failures	Errors	Time
1125	1 💤	0 ❌	0 🔥	23m 9s ⏱️

[daniel-herrero]

You missed a comma

[bameda]

ouch

[daniel-herrero]

Mega clap!