Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Section change error #504

Closed
wants to merge 75 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
7a13900
move SWC Sample parsing out of errorMessages header
mgeplf Feb 14, 2022
3912b6a
use braces in `if`s
mgeplf Apr 1, 2022
eadb1fa
add test for German morphologies
mgeplf May 2, 2023
727553f
force locale
mgeplf May 22, 2023
85a82f3
manual swc parser
mgeplf May 23, 2023
2926a02
see locales
mgeplf May 23, 2023
eeb5b0f
simplify logic
mgeplf May 23, 2023
9ebe396
see locales
mgeplf May 23, 2023
0bf9f84
add german locales to github linux installs
mgeplf May 23, 2023
52764ea
quell "error: macro is not used"
mgeplf May 23, 2023
c918315
reduce the creation and free'ing of locals
mgeplf May 23, 2023
e61140b
handle errors better
mgeplf May 23, 2023
51a836d
added trailing space test for swc; format
mgeplf May 24, 2023
abb8c13
switch to better interface
mgeplf May 24, 2023
ab275c6
cleanup macro
mgeplf May 24, 2023
fb89470
try Swiss German
mgeplf May 24, 2023
a3fa6f8
we should name locales the same
mgeplf May 24, 2023
fa0eb5c
cleanup includes
mgeplf May 24, 2023
62fa6e5
cleanup
mgeplf May 31, 2023
479679e
working
mgeplf Jun 20, 2023
6c39d49
added error for EARLY_END_OF_FILE
mgeplf Jun 20, 2023
756373c
cleanup
mgeplf Jun 20, 2023
5444ef3
docstrings
mgeplf Jun 20, 2023
d05acdc
Merge remote-tracking branch 'origin/master' into no-scanf-tokenstream
mgeplf Jun 22, 2023
a1c4dc1
update test
mgeplf Jun 22, 2023
e7a674a
better comment
mgeplf Jun 22, 2023
64428e1
Merge remote-tracking branch 'origin/master' into no-scanf-tokenstream
mgeplf Jun 23, 2023
347e356
coverage
mgeplf Jun 23, 2023
ce25568
Merge remote-tracking branch 'origin/master' into no-scanf-tokenstream
mgeplf Jun 29, 2023
fada478
format
mgeplf Jun 29, 2023
9464df9
use `at` for map access
mgeplf Jul 3, 2023
7657aab
Merge remote-tracking branch 'origin/master' into no-scanf-tokenstream
mgeplf Jul 3, 2023
00b8bbb
Merge remote-tracking branch 'origin/master' into no-scanf-tokenstream
mgeplf Jul 4, 2023
5355072
rename private members
mgeplf Jul 4, 2023
cf83223
make readSamples standalone
mgeplf Jul 4, 2023
6e07e4a
simplify building of SWC morphologies
mgeplf Jul 14, 2023
e3ff9fa
simplify buildSWC
mgeplf Jul 17, 2023
8366949
cleanup
mgeplf Jul 18, 2023
95a3fc1
* add test for test_multi_type_section
mgeplf Jul 25, 2023
18cc4f5
fix dendrites originating from non-first-soma sample
mgeplf Jul 26, 2023
ddcba7a
follow old version where bifurcated soma w/ 3 points was always SOMA_…
mgeplf Jul 27, 2023
cbcd3f2
check for missing parents
mgeplf Jul 27, 2023
93da395
soma-less root sections should include their own point
mgeplf Jul 27, 2023
889f5f5
clang format
mgeplf Jul 27, 2023
57e4e98
cleanup
mgeplf Jul 27, 2023
37206d7
properly handle windows end of line in swc parser
mgeplf Sep 15, 2023
24e0a52
style
mgeplf Oct 10, 2023
d8fcc4b
Merge remote-tracking branch 'origin/master' into efficient-swc-build
mgeplf Oct 10, 2023
2954e66
changelog
mgeplf Oct 10, 2023
a7e56ec
fix warning test
mgeplf Oct 10, 2023
5e3fa05
fix neuromorpho 3 point somas
mgeplf Oct 11, 2023
19f5e2d
fix test_root_node_split
mgeplf Oct 11, 2023
0457d8b
should apt-get update first
mgeplf Oct 16, 2023
701165c
fix missing data
mgeplf Oct 17, 2023
35d5a2d
Merge remote-tracking branch 'origin/master' into efficient-swc-build
mgeplf Jan 9, 2024
37c008c
asdf
mgeplf Jan 10, 2024
2688e33
fix 3 point soma handling
mgeplf Jan 12, 2024
4bd6718
Only H5/asc files require root points w/ two or more.
mgeplf Jan 12, 2024
9e12967
raise exception for writing empty morphology
mgeplf Jan 12, 2024
be13c21
add support for extra columns that are ignored
mgeplf Feb 13, 2024
a44b254
Merge remote-tracking branch 'origin/master' into efficient-swc-build
mgeplf May 14, 2024
e84d8b5
add support for extra columns that are ignored
mgeplf May 14, 2024
9b94928
fix
mgeplf May 14, 2024
0f15036
format
mgeplf May 15, 2024
216090e
asdf
mgeplf May 15, 2024
0a31446
add locale-de to coverage build
mgeplf May 15, 2024
ea58535
fix test
mgeplf May 15, 2024
a244da6
remove test_root_node_split
mgeplf May 15, 2024
7bdb495
fix typo
mgeplf Sep 10, 2024
f58f391
Create a warning if the section type changes without a bifurcation
mgeplf Sep 11, 2024
39c1311
Add option to allow for sections changes in SWC files without bifurca…
mgeplf Oct 4, 2024
51db41b
Merge remote-tracking branch 'origin/master' into section-change-error
mgeplf Oct 4, 2024
11da3be
debug
mgeplf Oct 4, 2024
85344eb
Merge remote-tracking branch 'origin/master' into section-change-error
mgeplf Oct 4, 2024
dac03be
Merge remote-tracking branch 'origin/master' into section-change-error
mgeplf Oct 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion binds/python/bind_enums.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ void bind_enums(py::module& m) {
.value("soma_sphere", morphio::enums::Option::SOMA_SPHERE)
.value("no_duplicates", morphio::enums::Option::NO_DUPLICATES)
.value("nrn_order", morphio::enums::Option::NRN_ORDER)
.value("unifurcated_section_change", morphio::enums::Option::UNIFURCATED_SECTION_CHANGE)
.export_values();


Expand All @@ -95,7 +96,8 @@ void bind_enums(py::module& m) {
.value("write_empty_morphology", morphio::enums::WRITE_EMPTY_MORPHOLOGY)
.value("zero_diameter", morphio::enums::Warning::ZERO_DIAMETER)
.value("soma_non_contour", morphio::enums::Warning::SOMA_NON_CONTOUR)
.value("soma_non_cylinder_or_point", morphio::enums::Warning::SOMA_NON_CYLINDER_OR_POINT);
.value("soma_non_cylinder_or_point", morphio::enums::Warning::SOMA_NON_CYLINDER_OR_POINT)
.value("type_changed_within_section", morphio::enums::Warning::SECTION_TYPE_CHANGED);

py::enum_<morphio::enums::SomaType>(m, "SomaType", py::arithmetic())
.value("SOMA_UNDEFINED", morphio::enums::SomaType::SOMA_UNDEFINED)
Expand Down
16 changes: 16 additions & 0 deletions binds/python/generated/docstrings.h
Original file line number Diff line number Diff line change
Expand Up @@ -770,6 +770,18 @@ static const char *mkd_doc_morphio_SectionBuilderError = R"doc()doc";

static const char *mkd_doc_morphio_SectionBuilderError_SectionBuilderError = R"doc()doc";

static const char *mkd_doc_morphio_SectionTypeChanged = R"doc()doc";

static const char *mkd_doc_morphio_SectionTypeChanged_SectionTypeChanged = R"doc()doc";

static const char *mkd_doc_morphio_SectionTypeChanged_errorLevel = R"doc()doc";

static const char *mkd_doc_morphio_SectionTypeChanged_lineNumber = R"doc()doc";

static const char *mkd_doc_morphio_SectionTypeChanged_msg = R"doc()doc";

static const char *mkd_doc_morphio_SectionTypeChanged_warning = R"doc()doc";

static const char *mkd_doc_morphio_Section_Section = R"doc()doc";

static const char *mkd_doc_morphio_Section_breadth_begin = R"doc(Breadth first iterator)doc";
Expand Down Expand Up @@ -1104,6 +1116,8 @@ static const char *mkd_doc_morphio_enums_Option_SOMA_SPHERE = R"doc(Interpret mo

static const char *mkd_doc_morphio_enums_Option_TWO_POINTS_SECTIONS = R"doc(Read sections only with 2 or more points)doc";

static const char *mkd_doc_morphio_enums_Option_UNIFURCATED_SECTION_CHANGE = R"doc(Allow section type to change without bifurcation)doc";

static const char *mkd_doc_morphio_enums_SectionType = R"doc(Classification of neuron substructures.)doc";

static const char *mkd_doc_morphio_enums_SectionType_SECTION_ALL = R"doc(Any section type)doc";
Expand Down Expand Up @@ -1211,6 +1225,8 @@ static const char *mkd_doc_morphio_enums_Warning_NO_SOMA_FOUND = R"doc(No soma f

static const char *mkd_doc_morphio_enums_Warning_ONLY_CHILD = R"doc(Single child sections are not allowed in SWC format)doc";

static const char *mkd_doc_morphio_enums_Warning_SECTION_TYPE_CHANGED = R"doc(In SWC, the type changed within a section, not post bifurcation)doc";

static const char *mkd_doc_morphio_enums_Warning_SOMA_NON_CONFORM = R"doc(Soma does not conform the three point soma spec from NeuroMorpho.org)doc";

static const char *mkd_doc_morphio_enums_Warning_SOMA_NON_CONTOUR = R"doc(Soma must be a contour for ASC and H5)doc";
Expand Down
1 change: 1 addition & 0 deletions doc/source/morphology.rst
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ The following flags are supported:
each section is no longer the last point of the parent section.
* ``morphio::NRN_ORDER``\: Neurite are reordered according to the
`NEURON simulator ordering <https://github.com/neuronsimulator/nrn/blob/2dbf2ebf95f1f8e5a9f0565272c18b1c87b2e54c/share/lib/hoc/import3d/import3d_gui.hoc#L874>`_
* ``morphio::UNIFURCATED_SECTION_CHANGE``\: Allow section type to change without bifurcation, emits warning

Multiple flags can be passed by using the standard bit flag manipulation (works the same way in C++
and Python):
Expand Down
4 changes: 3 additions & 1 deletion include/morphio/enums.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ enum Option {
TWO_POINTS_SECTIONS = 0x01, //!< Read sections only with 2 or more points
SOMA_SPHERE = 0x02, //!< Interpret morphology soma as a sphere
NO_DUPLICATES = 0x04, //!< Skip duplicating points
NRN_ORDER = 0x08 //!< Order of neurites will be the same as in NEURON simulator
NRN_ORDER = 0x08, //!< Order of neurites will be the same as in NEURON simulator
UNIFURCATED_SECTION_CHANGE = 0x10 //!< Allow section type to change without bifurcation
};

/**
Expand All @@ -42,6 +43,7 @@ enum Warning {
ZERO_DIAMETER, //!< Zero section diameter
SOMA_NON_CONTOUR, //!< Soma must be a contour for ASC and H5
SOMA_NON_CYLINDER_OR_POINT, //!< Soma must be stacked cylinders or a point
SECTION_TYPE_CHANGED, //!< In SWC, the type changed within a section, not post bifurcation
};

enum AnnotationType {
Expand Down
17 changes: 17 additions & 0 deletions include/morphio/warning_handling.h
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,23 @@ struct ZeroDiameter: public WarningMessage {
uint64_t lineNumber;
};

struct SectionTypeChanged: public WarningMessage {
SectionTypeChanged(std::string uri_, uint64_t lineNumber_)
: WarningMessage(std::move(uri_))
, lineNumber(lineNumber_) {}
morphio::enums::Warning warning() const final {
return Warning::SECTION_TYPE_CHANGED;
}
morphio::readers::ErrorLevel errorLevel = morphio::readers::ErrorLevel::WARNING;
std::string msg() const final {
static const char* description =
"Warning: Type changed within section, without bifurcation";
return "\n" + details::errorLink(uri, lineNumber, errorLevel) + description;
}

uint64_t lineNumber;
};

struct DisconnectedNeurite: public WarningMessage {
DisconnectedNeurite(std::string uri_, uint64_t lineNumber_)
: WarningMessage(std::move(uri_))
Expand Down
23 changes: 15 additions & 8 deletions src/readers/morphologySWC.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -234,14 +234,15 @@ class SWCBuilder
using Samples = std::vector<SWCSample>;

public:
SWCBuilder(std::string path, WarningHandler* warning_handler)
SWCBuilder(std::string path, WarningHandler* warning_handler, unsigned int options)
: path_(std::move(path))
, warning_handler_(warning_handler) {}
, warning_handler_(warning_handler)
, options_(options) {}

Property::Properties buildProperties(const std::string& contents, unsigned int options) {
Property::Properties buildProperties(const std::string& contents) {
const Samples samples = readSamples(contents, path_);
buildSWC(samples);
morph_.applyModifiers(options);
morph_.applyModifiers(options_);
return morph_.buildReadOnly();
}

Expand Down Expand Up @@ -477,7 +478,14 @@ class SWCBuilder
while (children_count == 1) {
sample = &samples_.at(id);
if(sample->type != samples_.at(children_.at(id)[0]).type){
break;
if (options_ & UNIFURCATED_SECTION_CHANGE) {
warning_handler_->emit(
std::make_unique<SectionTypeChanged>(path_, sample->lineNumber));
break;
}
throw RawDataError("Section type changed without a bifucation at line: " +
std::to_string(sample->lineNumber) +
", consider using UNIFURCATED_SECTION_CHANGE option");
}
points.push_back(sample->point);
diameters.push_back(sample->diameter);
Expand Down Expand Up @@ -518,9 +526,8 @@ class SWCBuilder
mut::Morphology morph_;
std::string path_;
WarningHandler* warning_handler_;
unsigned int options_;
};


} // namespace details

namespace readers {
Expand All @@ -530,7 +537,7 @@ Property::Properties load(const std::string& path,
unsigned int options,
std::shared_ptr<WarningHandler>& warning_handler) {
auto properties =
details::SWCBuilder(path, warning_handler.get()).buildProperties(contents, options);
details::SWCBuilder(path, warning_handler.get(), options).buildProperties(contents);

properties._cellLevel._cellFamily = NEURON;
properties._cellLevel._version = {"swc", 1, 0};
Expand Down
31 changes: 29 additions & 2 deletions tests/test_1_swc.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
strip_color_codes)

from morphio import (MorphioError, Morphology, RawDataError, SomaError,
SomaType, Warning, ostream_redirect, set_raise_warnings)
SomaType, Warning, ostream_redirect, set_raise_warnings, Option)


DATA_DIR = Path(__file__).parent / "data"
Expand Down Expand Up @@ -568,13 +568,36 @@ def test_throw_on_negative_id():
Morphology(content, extension='swc')


def test_axon_carrying_dendrite():
contents =('''
1 1 0 0 1 1 -1
2 2 0 0 2 2 1
3 2 0 0 3 3 2

4 3 0 0 4 4 3 # dendrite splits off
5 3 0 0 5 5 4

6 2 0 0 6 6 3 # axon carries on
7 2 0 0 7 7 3
''')
Morphology(contents, "swc")


def test_multi_type_section():
contents =('''1 1 0 4 0 3.0 -1
2 6 0 0 2 0.5 1 # <- type 6
3 7 0 0 3 0.5 2 # <- type 7
4 8 0 0 4 0.5 3 # <- type 8
5 9 0 0 5 0.5 4''') # <- type 9
n = Morphology(contents, "swc")

with pytest.raises(RawDataError):
Morphology(contents, "swc")

warnings = morphio.WarningHandlerCollector()
n = Morphology(contents,
"swc",
warning_handler=warnings,
options=Option.unifurcated_section_change)
assert_array_equal(n.soma.points, [[0, 4, 0]])
assert_array_equal(n.soma.diameters, [6.0])
assert len(n.root_sections) == 1
Expand All @@ -583,6 +606,10 @@ def test_multi_type_section():
np.array([[0, 0, 2], ]))
assert len(n.sections) == 4
assert_array_equal(n.section_offsets, [0, 1, 3, 5, 7])
warnings = [f.warning for f in warnings.get_all()]
assert len(warnings) == 3 # type 7, 8, and 9
for warning in warnings:
assert warning.warning() == Warning.type_changed_within_section


def test_missing_parent():
Expand Down
10 changes: 5 additions & 5 deletions tests/test_swc_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -130,15 +130,15 @@ TEST_CASE("morphio::swc::working") {
}

SECTION("chimera-axon-on-dendrite") {
const auto* no_soma = R"(
const auto* aod = R"(
1 1 0 0 1 1 -1
2 2 0 0 2 2 1
3 2 0 0 3 3 2
4 3 0 0 4 4 3
5 3 0 0 5 5 4
5 3 0 0 5 5 3
)";
const auto m = Morphology(no_soma, "swc");
REQUIRE(m.sections().size() == 2);
REQUIRE(m.diameters().size() == 5);
const auto m = Morphology(aod, "swc");
REQUIRE(m.sections().size() == 3);
REQUIRE(m.diameters().size() == 6);
}
}
Loading