Skip to content

Commit

Permalink
Make concept based table IDs 64 bit
Browse files Browse the repository at this point in the history
While we still have some headroom for the datasets we work with at the
moment, keeping the IDs of concept table based tables (inheritance) at
32 bit would have been a limit at some point. Already we see large
segmentation datasets that make this headroom rather small. Therefore
this migration will rewrite all concept based tables as well as tables
that reference them through foreign keys so that 64 bit IDs are used.

This migration also updates the review table to use 64 bit IDs. It also
fixes the history views of the catmaid_sampler table, which haven't been
kept up to date with column changes.

In addition a lot of missing foreign key constraints have been added.
They shouldn't have a big import on performance and in fact no real
difference could be measured.

The column order of concept changed slightly to improve the column
alignment and reduce padding. So far we wasted 4 Bytes per row, because
project_id occupied 4 Bytes and required 4 Bytes of padding, because it
was followed by a bigint. This is changed now and no more padding is
needed. This also lowers the impact on storage of the change to 64 bit
IDs.

Fixes #1848
  • Loading branch information
tomka committed Jun 22, 2019
1 parent 1cdb459 commit c29747e
Show file tree
Hide file tree
Showing 4 changed files with 2,602 additions and 1 deletion.
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ Contributors: Chris Barnes, Albert Cardona, Andrew Champion, Stephan Gerhard, Pa
Postgres superuser:
`sudo -u postgres psql -d <catmaid-db> -c 'CREATE EXTENSION pg_trgm;'`

- Due to changes in the database schema, migration 70 might take a while to
complete.

- CATMAID's version information changes back to a plain `git describe` format.
This results overall in a simpler setup and makes live easier for some
third-party front-ends, because the commit counter is included again. The
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Generated by Django 2.1.8 on 2019-05-29 12:46

from django.db import migrations


forward = """
UPDATE change_request
SET id = nextval('concept_id_seq');
ALTER TABLE ONLY change_request ALTER COLUMN id SET DEFAULT nextval('concept_id_seq'::regclass);
DROP SEQUENCE change_request_id_seq;
"""

backward = """
CREATE SEQUENCE change_request_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER SEQUENCE change_request_id_seq OWNED BY change_request.id;
ALTER TABLE ONLY change_request ALTER COLUMN id SET DEFAULT nextval('change_request_id_seq'::regclass);
UPDATE change_request
SET id = nextval('concept_id_seq');
"""

class Migration(migrations.Migration):
"""This removes the ID sequence the change_requests table is using and makes
it use the regular concept ID sequence (it inherits from concept). New IDs
will be assigned to all change requests. It is not possible to revert to the
original IDs.
"""

dependencies = [
('catmaid', '0070_remove_pg10_bug_workaround'),
]

operations = [
migrations.RunSQL(forward, backward)
]
Loading

0 comments on commit c29747e

Please sign in to comment.