Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slim dump/load implementation #146

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Slim dump/load implementation #146

wants to merge 5 commits into from

Conversation

Pliner
Copy link
Contributor

@Pliner Pliner commented Jan 15, 2024

TLDR: slim methods speed up transaction_load_dump.py from 0.019 seconds to 0.003 seconds.

Slim version:

 7067 function calls in 0.003 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      128    0.001    0.000    0.002    0.000 serialization_slim.py:291(load)
      128    0.000    0.000    0.001    0.000 serialization_slim.py:269(dump)
      256    0.000    0.000    0.000    0.000 serialization_slim.py:244(load)
      256    0.000    0.000    0.000    0.000 serialization_slim.py:236(dump)
      128    0.000    0.000    0.000    0.000 uuid.py:139(__init__)
      128    0.000    0.000    0.000    0.000 {method 'isoformat' of 'datetime.datetime' objects}
      128    0.000    0.000    0.000    0.000 <string>:2(__init__)
      128    0.000    0.000    0.000    0.000 uuid.py:280(__str__)
     1284    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
      128    0.000    0.000    0.000    0.000 serialization_slim.py:157(load)
      512    0.000    0.000    0.000    0.000 {method 'quantize' of 'decimal.Decimal' objects}
      128    0.000    0.000    0.000    0.000 serialization_slim.py:152(dump)
        1    0.000    0.000    0.002    0.002 serialization_slim.py:112(load)
        1    0.000    0.000    0.001    0.001 serialization_slim.py:93(dump)
      128    0.000    0.000    0.000    0.000 serialization_slim.py:216(load)
      128    0.000    0.000    0.000    0.000 serialization_slim.py:211(dump)
      642    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      384    0.000    0.000    0.000    0.000 {method 'replace' of 'str' objects}
      640    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
        1    0.000    0.000    0.003    0.003 {built-in method builtins.exec}
      256    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
      512    0.000    0.000    0.000    0.000 {method 'is_nan' of 'decimal.Decimal' objects}
      512    0.000    0.000    0.000    0.000 {method 'is_infinite' of 'decimal.Decimal' objects}
        1    0.000    0.000    0.003    0.003 <string>:1(<module>)
      128    0.000    0.000    0.000    0.000 {built-in method fromisoformat}
      128    0.000    0.000    0.000    0.000 {method 'count' of 'list' objects}
      128    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}
      130    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.001    0.001 serialization_slim.py:29(dump_slim_many)
        2    0.000    0.000    0.000    0.000 serialization_slim.py:328(__get_field_for_dataclass)
        2    0.000    0.000    0.000    0.000 dataclasses.py:1256(is_dataclass)
        1    0.000    0.000    0.002    0.002 serialization_slim.py:54(load_slim_many)
        2    0.000    0.000    0.000    0.000 <string>:2(__hash__)
        2    0.000    0.000    0.000    0.000 <string>:2(__eq__)
        2    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.hash}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Baseline:

 35400 function calls (35016 primitive calls) in 0.019 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    258/2    0.002    0.000    0.014    0.007 schema.py:576(_deserialize)
     1280    0.001    0.000    0.008    0.000 fields.py:344(deserialize)
      768    0.001    0.000    0.001    0.000 fields.py:1093(_format_num)
     1024    0.001    0.000    0.002    0.000 fields.py:265(_validate)
     1280    0.001    0.000    0.008    0.000 schema.py:655(getter)
     1024    0.001    0.000    0.001    0.000 fields.py:271(_validate_all)
      640    0.001    0.000    0.003    0.000 fields.py:317(serialize)
      256    0.001    0.000    0.001    0.000 uuid.py:139(__init__)
      512    0.001    0.000    0.002    0.000 fields.py:1103(_validated)
    129/1    0.001    0.000    0.004    0.004 schema.py:501(_serialize)
     1280    0.001    0.000    0.009    0.000 schema.py:481(_call_and_store)
        2    0.000    0.000    0.014    0.007 schema.py:611(<listcomp>)
      640    0.000    0.000    0.001    0.000 utils.py:262(get_value)
      256    0.000    0.000    0.001    0.000 inspect.py:391(_has_code_flag)
     2562    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
     1024    0.000    0.000    0.000    0.000 validate.py:66(__init__)
     1280    0.000    0.000    0.000    0.000 utils.py:298(set_value)
      640    0.000    0.000    0.000    0.000 utils.py:288(_get_value_for_key)
        4    0.000    0.000    0.001    0.000 schema.py:1221(<listcomp>)
     1024    0.000    0.000    0.000    0.000 validate.py:73(__call__)
      256    0.000    0.000    0.001    0.000 fields.py:912(_validated)
      768    0.000    0.000    0.000    0.000 {method 'quantize' of 'decimal.Decimal' objects}
      256    0.000    0.000    0.001    0.000 fields.py:673(_deserialize)
      128    0.000    0.000    0.000    0.000 {method 'isoformat' of 'datetime.datetime' objects}
     1280    0.000    0.000    0.000    0.000 fields.py:308(_validate_missing)
      640    0.000    0.000    0.001    0.000 fields.py:253(get_value)
      512    0.000    0.000    0.001    0.000 fields.py:952(_validated)
      512    0.000    0.000    0.002    0.000 fields.py:976(_deserialize)
      640    0.000    0.000    0.001    0.000 schema.py:469(get_attribute)
      128    0.000    0.000    0.000    0.000 uuid.py:280(__str__)
      256    0.000    0.000    0.000    0.000 fields.py:1285(_deserialize)
      256    0.000    0.000    0.001    0.000 utils.py:46(is_generator)
     1670    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      256    0.000    0.000    0.001    0.000 fields.py:969(_serialize)
      258    0.000    0.000    0.001    0.000 utils.py:51(is_iterable_but_not_string)
      768    0.000    0.000    0.000    0.000 {method 'replace' of 'str' objects}
      256    0.000    0.000    0.000    0.000 fields.py:681(_serialize)
      258    0.000    0.000    0.001    0.000 utils.py:56(is_collection)
      256    0.000    0.000    0.001    0.000 inspect.py:402(isgeneratorfunction)
      256    0.000    0.000    0.001    0.000 fields.py:925(_deserialize)
      900    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}
      128    0.000    0.000    0.000    0.000 bake.py:287(<dictcomp>)
      256    0.000    0.000    0.000    0.000 inspect.py:378(isfunction)
      128    0.000    0.000    0.000    0.000 fields.py:1276(_serialize)
      128    0.000    0.000    0.000    0.000 bake.py:290(post_load)
      257    0.000    0.000    0.000    0.000 typing.py:359(inner)
      256    0.000    0.000    0.000    0.000 inspect.py:2077(_signature_is_functionlike)
      256    0.000    0.000    0.000    0.000 functools.py:421(_unwrap_partial)
      128    0.000    0.000    0.000    0.000 bake.py:284(remove_none_values)
      256    0.000    0.000    0.000    0.000 inspect.py:300(ismethod)
      128    0.000    0.000    0.000    0.000 utils.py:242(ensure_text_type)
      258    0.000    0.000    0.000    0.000 <frozen abc>:117(__instancecheck__)
      256    0.000    0.000    0.000    0.000 inspect.py:428(isgenerator)
      256    0.000    0.000    0.000    0.000 {built-in method builtins.format}
      768    0.000    0.000    0.000    0.000 {method 'is_finite' of 'decimal.Decimal' objects}
      256    0.000    0.000    0.000    0.000 bake.py:294(pre_load)
      384    0.000    0.000    0.000    0.000 schema.py:414(dict_class)
      256    0.000    0.000    0.000    0.000 fields.py:1113(_to_string)
      644    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
      513    0.000    0.000    0.000    0.000 typing.py:2268(cast)
      258    0.000    0.000    0.000    0.000 {built-in method _abc._abc_instancecheck}
      256    0.000    0.000    0.000    0.000 {built-in method fromisoformat}
      128    0.000    0.000    0.000    0.000 fields.py:892(_serialize)
        2    0.000    0.000    0.014    0.007 schema.py:811(_do_load)
      512    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
      512    0.000    0.000    0.000    0.000 {built-in method builtins.callable}
      512    0.000    0.000    0.000    0.000 {method 'is_infinite' of 'decimal.Decimal' objects}
      512    0.000    0.000    0.000    0.000 {method 'is_nan' of 'decimal.Decimal' objects}
        1    0.000    0.000    0.004    0.004 schema.py:512(<listcomp>)
      256    0.000    0.000    0.000    0.000 {method 'count' of 'list' objects}
      256    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}
        1    0.000    0.000    0.019    0.019 {built-in method builtins.exec}
      256    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.019    0.019 <string>:1(<module>)
        1    0.000    0.000    0.004    0.004 schema.py:525(dump)
        1    0.000    0.000    0.007    0.007 schema.py:779(validate)
        1    0.000    0.000    0.012    0.012 serialization.py:91(dump_many_v3)
        8    0.000    0.000    0.001    0.000 schema.py:1196(_invoke_processors)
        3    0.000    0.000    0.001    0.000 schema.py:1077(_invoke_load_processors)
        2    0.000    0.000    0.000    0.000 serialization.py:57(schema_v3)
        1    0.000    0.000    0.007    0.007 serialization.py:73(load_many_v3)
        2    0.000    0.000    0.000    0.000 schema.py:1106(_invoke_field_validators)
        1    0.000    0.000    0.000    0.000 schema.py:1063(_invoke_dump_processors)
        2    0.000    0.000    0.000    0.000 <string>:2(__init__)
        1    0.000    0.000    0.007    0.007 schema.py:692(load)
        2    0.000    0.000    0.000    0.000 <string>:2(__hash__)
        2    0.000    0.000    0.000    0.000 <string>:2(__eq__)
        2    0.000    0.000    0.000    0.000 error_store.py:13(__init__)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.hash}

@Pliner Pliner requested a review from a team as a code owner January 15, 2024 07:57
@Pliner Pliner changed the title Slim version of marshmallow Slim dump/load implementation Jan 15, 2024
outring
outring reviewed Jan 19, 2024
View reviewed changes
marshmallow_recipe/serialization_slim.py

def dump(self, value: Any) -> Any:
if not isinstance(value, list):
raise m.ValidationError("Not a valid list.")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
raise m.ValidationError("Not a valid list.")
raise m.ValidationError("Not a valid list")

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And in all other places as well

marshmallow_recipe/serialization_slim.py
try:
return uuid.UUID(value)
except ValueError:
raise m.ValidationError("Not a valid uuid.")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would've been nice to see an actual value in the exception 🤔

marshmallow_recipe/serialization_slim.py
error_messages[dataclass_field.name] = field_validation_error.messages

if error_messages is not None:
raise m.ValidationError(error_messages, NO_FIELD_NAME)
Copy link

@outring outring Jan 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You plan to add a parent field name at some point instead of NO_FIELD_NAME, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@outring outring outring left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Pliner @outring