-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use Lark with its cache feature, instead of creating a standalone parser #53
Merged
Merged
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
5b81783
Use Lark with its cache feature, instead of creating a standalone parser
erezsh 726bf0c
Restore cache (accidentally left disabled)
erezsh c1c5cd9
fix lint errors
aoskotsky-amplify 9961e0c
Merge with master.
htorianik-amplify b31d9ec
Refactor parser.py.
htorianik-amplify 37e5f2b
Add explanation why we ignore typing in api.py.
htorianik-amplify File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -117,3 +117,4 @@ node_modules/ | |
|
||
# Don't commit the generated parser | ||
lark_parser.py | ||
.lark_cache.bin |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,53 +1,17 @@ | ||
"""A parser for HCL2 implemented using the Lark parser""" | ||
import os | ||
from os.path import exists, dirname | ||
from os.path import dirname | ||
|
||
from lark import Lark | ||
from lark.grammar import Rule | ||
from lark.lexer import TerminalDef | ||
|
||
from hcl2.transformer import DictTransformer | ||
|
||
PARSER_FILE = os.path.join(dirname(__file__), 'lark_parser.py') | ||
PARSER_FILE = os.path.join(dirname(__file__), '.lark_cache.bin') | ||
|
||
PARSER_FILE_TEMPLATE = """ | ||
from lark import Lark | ||
|
||
DATA = (%s) | ||
MEMO = (%s) | ||
|
||
def Lark_StandAlone(**kwargs): | ||
return Lark._load_from_dict(DATA, MEMO, **kwargs) | ||
""" | ||
|
||
|
||
def create_parser_file(): | ||
""" | ||
Parsing the Lark grammar takes about 0.5 seconds. In order to improve performance we can cache the parser | ||
file. The below code caches the entire python file which is generated by Lark's standalone parser feature | ||
See: https://github.com/lark-parser/lark/blob/master/lark/tools/standalone.py | ||
|
||
Lark also supports serializing the parser config but the deserialize function did not work for me. | ||
The lark state contains dicts with numbers as keys which is not supported by json so the serialized | ||
state can't be written to a json file. Exporting to other file types would have required | ||
adding additional dependencies or writing a lot more code. Lark's standalone parser | ||
feature works great but it expects to be run as a separate shell command | ||
The below code copies some of the standalone parser generator code in a way that we can use | ||
""" | ||
lark_file = os.path.join(dirname(__file__), 'hcl2.lark') | ||
with open(lark_file, 'r') as lark_file, open(PARSER_FILE, 'w') as parser_file: | ||
lark_inst = Lark(lark_file.read(), parser="lalr", lexer="standard") | ||
|
||
data, memo = lark_inst.memo_serialize([TerminalDef, Rule]) | ||
|
||
print(PARSER_FILE_TEMPLATE % (data, memo), file=parser_file) | ||
|
||
|
||
if not exists(PARSER_FILE): | ||
create_parser_file() | ||
|
||
# pylint: disable=wrong-import-position | ||
# Lark_StandAlone needs to be imported after the above block of code because lark_parser.py might not exist | ||
from hcl2.lark_parser import Lark_StandAlone | ||
|
||
hcl2 = Lark_StandAlone(transformer=DictTransformer()) | ||
hcl2 = Lark.open( | ||
'hcl2.lark', | ||
parser='lalr', | ||
cache=PARSER_FILE, # Disable/Delete file to effect changes to the grammar | ||
rel_to=__file__, | ||
transformer=DictTransformer() | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,3 @@ | ||
"""Place of record for the package version""" | ||
|
||
__version__ = "2.0.1" | ||
__version__ = "2.1.0" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,3 @@ | ||
# Place dependencies in this file, following the distutils format: | ||
# http://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages | ||
lark-parser>=0.10.0,<0.11.0 | ||
lark-parser>=0.11.0,<0.12.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mypy
was complaining thatparse
returns aTree
instead of aDict
. We can leave the ignore here but was curious why it does that.