-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[move-compiler-v2] add parser code for lambda types #14792
base: main
Are you sure you want to change the base?
Conversation
⏱️ 2h 25m total CI duration on this PR
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 09-27-extend_parser_for_lambda_types #14792 +/- ##
=======================================================================
Coverage ? 59.9%
=======================================================================
Files ? 852
Lines ? 207826
Branches ? 0
=======================================================================
Hits ? 124528
Misses ? 83298
Partials ? 0
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
7f59256
to
ae73f37
Compare
dbd482d
to
270214e
Compare
ae73f37
to
243ea0f
Compare
270214e
to
afbad9b
Compare
243ea0f
to
f3d30eb
Compare
a40a3c9
to
71ef91f
Compare
f3d30eb
to
0cc6890
Compare
71ef91f
to
3a1d362
Compare
0cc6890
to
1b54f30
Compare
c46a244
to
b3c1106
Compare
ad600f0
to
017f471
Compare
b3c1106
to
a65ee0f
Compare
017f471
to
b7d429a
Compare
a65ee0f
to
cbb77b0
Compare
b7d429a
to
c99de22
Compare
cbb77b0
to
b2185c2
Compare
c99de22
to
d3254ad
Compare
b2185c2
to
3f98e56
Compare
d3254ad
to
7e21319
Compare
3f98e56
to
e24bf4e
Compare
7e21319
to
7cfe2c4
Compare
e24bf4e
to
2a740e8
Compare
2a740e8
to
dabc427
Compare
third_party/move/move-compiler-v2/tests/bytecode-generator/freeze_mut_ref.exp
Outdated
Show resolved
Hide resolved
@@ -215,6 +216,10 @@ fn check_privileged_operations_on_structs(env: &GlobalEnv, fun_env: &FunctionEnv | |||
| Operation::MoveFrom | |||
| Operation::MoveTo => { | |||
let inst = env.get_node_instantiation(*id); | |||
if inst.is_empty() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these debug prints will be removed later I believe?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, debug prints can stay but they should be protected by a module flag DEBUG
which is off when submitted, so people debugging other parts of the compiler are not bothered by this.
Those debug prints however look like temporary and should be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps. Note that these will only be printed if the following assert will fail, so I'm leaving them in for now for debugging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to remove this before landing
third_party/move/move-compiler/tests/move_check/parser/invalid_call_lhs_complex_expression.exp
Show resolved
Hide resolved
third_party/move/move-compiler-v2/tests/lambda/storable/doable_func.move
Outdated
Show resolved
Hide resolved
third_party/move/move-compiler-v2/tests/lambda/storable/registry.move
Outdated
Show resolved
Hide resolved
@@ -2930,17 +2982,18 @@ impl<'env> ModuleEnv<'env> { | |||
return deps; | |||
} | |||
for fun_env in self.get_functions() { | |||
let called_funs = fun_env.get_called_functions().expect("called functions"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When should we use used_functions
instead of called_functions
and vice versa?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually suspect that used_functions
should subsume called_functions
but there are bunch of calls under move-prover
which I'm going to need help with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This came out more complex than I was hoping for. Major ask is to reduce back to the existing expression variants (use Operation::Closure
which already existed), and change the syntax to |T| has A
instead of colon. I wish we would not have done the f(_)
for the MVP, this creates significant complexity all over the place, and I'm not sure whether I can check correctness of all those changes just from this PR.
aptos-move/framework/aptos-framework/tests/compiler-v2-doc/account.md
Outdated
Show resolved
Hide resolved
@@ -786,10 +786,13 @@ impl AbilitySet { | |||
| (Ability::Store as u8) | |||
| (Ability::Key as u8), | |||
); | |||
/// Abilities for user-defined/"primitive" functions (not closures) | |||
pub const DEFINED_FUNCTIONS: AbilitySet = | |||
Self((Ability::Copy as u8) | (Ability::Drop as u8) | (Ability::Store as u8)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Store doesn't seem to be right.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After our last discussion, yes, it should only be Store if it's public.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
Lambda(NodeId, Pattern, Exp), | ||
Lambda(NodeId, Pattern, Exp, LambdaCaptureKind, AbilitySet), | ||
/// Represents a reference to a Move Function as a function value. | ||
MoveFunctionExp(NodeId, ModuleId, FunId), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The philosophy of the exp language to keep it as small as possible. We do not need this thing, it can be folded into 'Curry', and Curry
can be renamed to make it more neutral, like Closure
. A closure can capture 0 arguments, why not.
We furthermore do not and should not have a new ExpData variant for it. The current design uses Call(..Operation..) where possible, and so should this. There is already the Closure operation, you can simply extend it
Call(_, Operation::Closure(f, mask), vec[captured])
This is rather important to me. I took great care to design ExpData like it is today, and it benefited us already many times. Lots of the code changes you did in visitors and elsewhere should become unnecessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have 2 distinct expressions we need to build:
- create a function value from a defined function by
(ModuleId, FunId)
- create a function value from a function value and 0 or more parameters
YourClosure
operation doesn't handle the latter case, as the "function value" need not be a reference to a defined function (ModuleId, FunId
) but can be an arbitrary expression.
If f
in Operation::Closure(f, mask)
can refer to an expression, then we have bigger consistency problems.
We might conceivably have Operation::Closure(mask)
with the function value as the first argument to the Call
expression, though it's a bit irregular.
OTOH, we might only allow directly creating a closure based on a Move function. This is more limiting, in that |x| f(x)
can only be used in MVP if f
is a user function, not if it's a function value. I guess perhaps that is what you had in mind. I prefer more symmetry.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've minimized the changes to ExpData
by adding Function
as a Value
, and instead of Closure
or Curry
I have an Operation
called Bind
, which takes a function expr and the list of parameters, with the mask as a field in the variant: Operation::Bind(mask)
.
Hopefully this is more to your liking, although since i decided to call it Bind
I realized we have Bind
in a few places. Perhaps I can rename it as EarlyBind
or something.
Lambda(NodeId, Pattern, Exp, LambdaCaptureKind, AbilitySet), | ||
/// Represents a reference to a Move Function as a function value. | ||
MoveFunctionExp(NodeId, ModuleId, FunId), | ||
/// Represents a Curry expression. mask has a 1 bit for position of each delayed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is more logical to set a bit if a parameter is captured and not the opposite.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the current rep, total args = bitcount(mask) + actual args.
With the alternative you suggest, actual args = bitcount(mask), and we need to check the function type to know how many args to expect in total.
True, we need to consult the function type to check the types of remaining args, but we get more information out of this rep.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After some reflection, I agree with you that using a 1 bit here for each bound parameter makes sense here.
if let Type::Fun(argt, _) = &ty { | ||
if argt.deref().has_function() { | ||
if let Type::Fun(args, _, _) = &ty { | ||
if args.deref().has_function() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deref is a trait function for *
and not idiomatic to use, as_ref
should be preferred.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
let fun_type = self.check_type(loc, &fun_type, expected_type, context); | ||
|
||
let id = self.parent.parent.env.new_node(loc.clone(), fun_type); | ||
return ExpData::MoveFunctionExp(id, module_id, fun_id); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be Call(id, Closure(module_id, fund_id, 0x0), vec![])
Reduce the number of variants in the IR is a key to maintainable and correct code.
@@ -3778,7 +3948,9 @@ impl<'env, 'translator, 'module_translator> ExpTranslator<'env, 'translator, 'mo | |||
.struct_table | |||
.contains_key(&global_var_sym) | |||
{ | |||
self.check_language_version(loc, "resource indexing", LanguageVersion::V2_0); | |||
if !self.check_language_version(loc, "resource indexing", LanguageVersion::V2_0) { | |||
return None; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait a minute -- why are we returning None here? (And the comment above?) This generates an error and this is all we need. The program is consistent independent of Move language version. This change here and elsewhere seems unnecessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, it returns more quickly rather than building pointless expressions and cascading errors if the feature is not supported. Here if we continue we might generate an additional error if the struct type used does not have key
ability.
To minimize changes to the code, instead I will change check_language_version
to return Option<()>
and return None
if the test fails, allowing use of ?
to more concisely bail out of the enclosing functions with a None
value in most cases.
if let Type::Reference(_, inner_ty) = ty { | ||
self.new_node_id_with_type_loc( | ||
&Type::Reference(ReferenceKind::Mutable, inner_ty), | ||
if mask != 0u128 && !matches!(cand, AnyFunEntry::UserFun(_)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A comment what you are doing here would be helpful. What magic number is 0u128?
I'm worried about this change and the complexity of that we support _
. The MVP doesn't need _
, it would have been SO much easier to leave the type checking mostly the same, and just derive the closure from the lambda without new function during lifting, after type checking. We would be already working at VM part.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, but someone thought it was a good idea originally. :-)
fun_exp_or_oper: FunExpOrOper, | ||
arg_types: Vec<Type>, | ||
translated_args: Vec<Exp>, | ||
mask: u128, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the mask for closures, it should be only u64. More is not needed. u128 should not be put into an instruction, to large for it, but its not needed anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sound good. Also check for max 64 params in lambda_lifter.rs
. I'm pretty sure that's limited elsewhere, but best to be sure.
}; | ||
let body = Box::new(parse_exp(context)?); | ||
let abilities_start = context.tokens.start_loc(); | ||
let abilities = parse_abilities(context)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The abilities of a function type should use |T|R with A
and not :
. It looks strange to have two colons in an argument declaration, as in f(x: |T|:copy
. Thats also the reason why it is not |T|:R
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I'm changing the syntax (for both lambda expression and function type) to "with copy+store
". This avoids a comma-separated list if we do "has copy, store
" (as in struct
decls) which would lead to parsing confusion in the case of expression or parameter lists.
5e47d93
to
626b1ed
Compare
626b1ed
to
f3901b3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I rolled back the _
support for now, due to persisting type/ability resolution issues. Enjoy.
@@ -2930,17 +2982,18 @@ impl<'env> ModuleEnv<'env> { | |||
return deps; | |||
} | |||
for fun_env in self.get_functions() { | |||
let called_funs = fun_env.get_called_functions().expect("called functions"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually suspect that used_functions
should subsume called_functions
but there are bunch of calls under move-prover
which I'm going to need help with.
third_party/move/move-compiler-v2/tests/lambda/storable/doable_func.move
Outdated
Show resolved
Hide resolved
@@ -891,6 +941,7 @@ impl<'env, 'translator, 'module_translator> ExpTranslator<'env, 'translator, 'mo | |||
self.translate_hlir_base_types(&args[0..args.len() - 1]), | |||
)), | |||
Box::new(self.translate_hlir_base_type(&args[args.len() - 1])), | |||
AbilitySet::FUNCTIONS, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AbilitySet::FUNCTIONS
is currently just drop
and copy
. Maybe will get rid of copy
, so we can allow closures with captured free variables with no copy
.
}; | ||
let body = Box::new(parse_exp(context)?); | ||
let abilities_start = context.tokens.start_loc(); | ||
let abilities = parse_abilities(context)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I'm changing the syntax (for both lambda expression and function type) to "with copy+store
". This avoids a comma-separated list if we do "has copy, store
" (as in struct
decls) which would lead to parsing confusion in the case of expression or parameter lists.
Lambda(NodeId, Pattern, Exp, LambdaCaptureKind, AbilitySet), | ||
/// Represents a reference to a Move Function as a function value. | ||
MoveFunctionExp(NodeId, ModuleId, FunId), | ||
/// Represents a Curry expression. mask has a 1 bit for position of each delayed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After some reflection, I agree with you that using a 1 bit here for each bound parameter makes sense here.
Lambda(NodeId, Pattern, Exp), | ||
Lambda(NodeId, Pattern, Exp, LambdaCaptureKind, AbilitySet), | ||
/// Represents a reference to a Move Function as a function value. | ||
MoveFunctionExp(NodeId, ModuleId, FunId), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've minimized the changes to ExpData
by adding Function
as a Value
, and instead of Closure
or Curry
I have an Operation
called Bind
, which takes a function expr and the list of parameters, with the mask as a field in the variant: Operation::Bind(mask)
.
Hopefully this is more to your liking, although since i decided to call it Bind
I realized we have Bind
in a few places. Perhaps I can rename it as EarlyBind
or something.
4745b14
to
9b3a2af
Compare
pub const FUNCTIONS: AbilitySet = Self(Ability::Drop as u8); | ||
/// Maximal abilities for all `Functions`. This is used for UB when joining types. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does UB
mean?
@@ -864,6 +871,8 @@ impl AbilitySet { | |||
pub const VECTOR: AbilitySet = | |||
Self((Ability::Copy as u8) | (Ability::Drop as u8) | (Ability::Store as u8)); | |||
|
|||
/// The empty ability set |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this comment for singleton
?
@@ -215,6 +216,10 @@ fn check_privileged_operations_on_structs(env: &GlobalEnv, fun_env: &FunctionEnv | |||
| Operation::MoveFrom | |||
| Operation::MoveTo => { | |||
let inst = env.get_node_instantiation(*id); | |||
if inst.is_empty() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to remove this before landing
@@ -1739,6 +1814,12 @@ impl ExpRewriterFunctions for LoopNestRewriter { | |||
#[derive(Debug, Clone, PartialEq, Eq, Hash)] | |||
pub enum Operation { | |||
MoveFunction(ModuleId, FunId), | |||
/// Build a closure by binding 1 or more leading arguments to a function value. | |||
/// First argument to the operation must be a function; the reamining |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: reaming -> remaining
…le of function type
…unnecessarily; avoid generating errors/warnings about default Loc
…ionExpr to Value::Function, got things mostly back in shape
…for abilities in lambda_lifter.rs
…constraints to avoid reducing set of function type abilities when unifying types
…to bind leading arguments
9395a8f
to
cead217
Compare
Description
Extend syntax
move |args| body with copy, store
|T1, T2| T3 with copy+store
|a, b| f(x, 3, a, b)
EarlyBind
expression now takes a function-valued expression and a set of argument expressions, which correspond to a prefix of the function parameters.(|x, y| x + y)(2, 3)
Modify exp_builder and lambda_lifter to generate function values.
Modify model to track "used" functions in addition to "called" functions to be able to catch all dependencies when function values are created but not called.
Attaches an
AbilitySet
toType::Fun
andExp::Lambda
based on source. Adds a newExpCall
operation in parser/expansion ASTs to be able to carry more generalized function calls through to move-model, which already can support this throughInvoke
, which previously was underutilized. Added basic type checking for function abilities.Current Gaps
I was targeting the test cases
return_func.move
anddoable_func.move
, but this PR is still missingstore
for function values built by curry with only storable parametersI will try to patch those in while this is being reviewed, or add them to a later PR.
How Has This Been Tested?
Added more lambda tests under
move-compiler-v2/tests/lambda/
which are run "with" and "without" lambda features enabled. Currently, many things pass through to hit "not yet implemented" errors in bytecode gen, etc.Ran and carefully checked all tests under third_party and aptos.
Key Areas to Review
Key features are illustrated in test
move-compiler-v2/tests/lambda/storable/doable_func.move
and the two corresponding output files, along with the testreturn_func.move
in the same directory. Most other test outputs should be largely unchanged, although error messages related to lambda use in default configuration have changed.Tricky features are:
ty.rs
: unification checksFun
abilities as well as other things.lambda_lifter.rs
, we (1) reject lambdas withoutmove
free-var handling, (2) try to reduce lambda to curry, by checking for a simple function call with simple args, the last of which are identical to the lambda parameters.Type of Change
Which Components or Systems Does This Change Impact?
Checklist