-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rewrite julia_to_gap
#1029
rewrite julia_to_gap
#1029
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1029 +/- ##
==========================================
+ Coverage 75.61% 75.81% +0.19%
==========================================
Files 55 55
Lines 4478 4531 +53
==========================================
+ Hits 3386 3435 +49
- Misses 1092 1096 +4
|
@fingolfin If it is of interest:
The log looks similar to the one from the failed CI tests on github (Julia 1.10 - ubuntu-latest), but I get the crash a bit later: The GAP 4.13.1 can load its packages, processes a few testfiles, and crashes in |
src/julia_to_gap.jl
Outdated
recursion_dict[obj] = ret_val | ||
end | ||
for i = 1:rows | ||
ret_val[i] = julia_to_gap(obj[i, :], recursion_dict; recursive) | ||
ret_val[i] = julia_to_gap_internal(obj[i, :], recursion_dict, recursive) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Existing issue, but shouldn't this also do the get!(recursion_dict::RecDict, x)
dance?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think no.
The idea of the get!
call is twofold.
- Check whether the object in question is already in the dictionary, and if yes then take the value.
- If the object is not yet in the dictionary then store it together with the converted value.
Situation 1. cannot occur because we are just now creating the Julia object anew.
And situation 2. would not help later on because no pointer to this newly created object can be found in the recursion.
(I will add a comment.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel there is something inconsistent here. Because then the object produced by recurively called julia_to_gap_internal
is potentially never added to recursion_dict
, is it? So if we had T = BigInt
we would not be caching the conversion results for numerator or denominator. While if we convert a list of BigInt
they would be cached.
Perhaps we never want to cache a BigInt
's converted while. But then why does the julia_to_gap_internal
method for Rational{T}
provide the the full 3-argument form instead of just using GAP.@install
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK the crucial point I missed is that we convert row-wise, i.e. we convert obj[i, :]
and not elementwise, i.e. obj[i,j]
src/julia_to_gap.jl
Outdated
recursion_dict[obj] = ret_val | ||
for i = 1:len | ||
ret_val[i] = julia_to_gap(obj[i], recursion_dict; recursive) | ||
ret_val[i] = julia_to_gap_internal(obj[i], recursion_dict, recursive) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another case with missing get!(recursion_dict::RecDict, x)
etc.?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, good spot!
I will add a test for this situation (a GAP list containing identical Julia objects which themselves do not support/need recursion).
src/julia_to_gap.jl
Outdated
recursive::Bool, | ||
) where {T} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could enhance the non-recursive case by using a Val
type instead:
recursive::Bool, | |
) where {T} | |
::Val{recursive}, | |
) where {T, recursive} |
This way the Julia compiler can produce seperate optimized code for both recursive = true
and recursive = false
.
src/julia_to_gap.jl
Outdated
) where {T} | ||
|
||
if recursion_dict !== nothing && haskey(recursion_dict, obj) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I wonder if this should be
if recursion_dict !== nothing && haskey(recursion_dict, obj) | |
if recursion && recursion_dict !== nothing && haskey(recursion_dict, obj) |
That is, I think the invariant "recursion_dict !== nothing implies recursion == true" should hold.
So this change normally shouldn't matter, but if we also change recursion
to use a Val
type then this will allow the compiler to elide this entire if
block if recursion == false
holds
src/julia_to_gap.jl
Outdated
recursion_dict[obj] = ret_val | ||
end | ||
for i = 1:rows | ||
ret_val[i] = julia_to_gap(obj[i, :], recursion_dict; recursive) | ||
ret_val[i] = julia_to_gap_internal(obj[i, :], recursion_dict, recursive) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel there is something inconsistent here. Because then the object produced by recurively called julia_to_gap_internal
is potentially never added to recursion_dict
, is it? So if we had T = BigInt
we would not be caching the conversion results for numerator or denominator. While if we convert a list of BigInt
they would be cached.
Perhaps we never want to cache a BigInt
's converted while. But then why does the julia_to_gap_internal
method for Rational{T}
provide the the full 3-argument form instead of just using GAP.@install
?
src/julia_to_gap.jl
Outdated
@@ -132,79 +140,83 @@ function julia_to_gap( | |||
continue | |||
end | |||
if recursive | |||
x = get!(recursion_dict, x) do | |||
julia_to_gap(x, recursion_dict; recursive) | |||
res = get!(recursion_dict::RecDict, x) do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to record some of my comments from our oral discussion earlier for posterity:
We should remove the get!
calls here and instead leave to each julia_to_gap_internal
method whether to use it (or not).
The idea here is: for e.g. Int16
we never want to use the cache. But for a more complex object we might.
For "simple" conversions, we can take care of this by adding a variant of GAP.@install
, say GAP.@install_rec
(just a provisional name; we could also rename the current one to GAP.@install_leaf
/GAP.@install_no_recursion
/GAP.@install_basic
/... and call the new one GAP.@install
)
The new one would roughly convert
GAP.@install_rec GapObj(obj::MyType) = foobar(obj)
to something like this
function julia_to_gap_internal(obj::T, recursion_dict::Nothing,
recursion::Bool)
return foobar(obj)
end
function julia_to_gap_internal(obj::T, recursion_dict::RecDict,
recursion::Bool)
@assert recursion
return get!(recursion_dict, obj) do
return foobar(obj)
end::Obj # TODO: is that right?
end
Note that in general we can have recursion == true && recursion_dict === nothing
; but it is up to any containers to allocate a recursion_dict
if necessary. We don't do it here, meaning that julia_to_gap(123 ; recursive=true)
just won't allocate a recursion dictionary, which is fine.
Regarding the type assertion on the get!
return value: I think it must always be a GAP.Obj
here? Perhaps we could even tighten it to GAP
if we agree to just not store Int
, Bool
, and FFE
values in recursion_dict
(but then the conversion of BigInt
needs some special casing perhaps... unless we decide to just never cache that one, then this problem essentially goes away, I think)
src/julia_to_gap.jl
Outdated
len = length(obj) | ||
ret_val = NewPlist(len) | ||
if recursive | ||
if recursion_dict === nothing | ||
recursion_dict = RecDict() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another optimization note for future iterations on this: if we are given e.g. a Vector{Int16}
and asked to recursively convert it to a GapObj
, then we really don't need the recursion_dict
, as there is no point storing the Int16
values in it.
To get rid of such things, my "dual" code in PR #777 does this:
if rec_dict === nothing && _needs_conversion_tracking(T)
rec_dict = RecDict()
end
where _needs_conversion_tracking(T)
returns true
if T
is a type where the conversion results should be tracked.
Such an approach interacts nicely with my suggestion to "split" GAP.@install
:
GAP.@install
(or whatever it is called) could also produce a_needs_conversion_tracking
method returningfalse
GAP.@install_rec
(or whatever it is called) could also produce a_needs_conversion_tracking
method returningtrue
Note: I guess we should have two different _needs_conversion_tracking
functions, with different names, for the two direction: to and from Julia. Perhaps they'll end up identical but this is not a priori clear to me? Anyway there would be no harm in having two for the moment.
b2806ff
to
b9acad3
Compare
The current test failures are due to crashes (Julia 1.11, Julia nightly) and due to a problem with As for the latter, one really gets a |
For the 1.6 failure: The place in AbstractAlgebra where the macroexpand idea came from limits to some julia versions. See https://github.com/Nemocas/AbstractAlgebra.jl/blob/025eabc50948f7db78bd5561acfb73bd5b340cc8/test/Attributes-test.jl#L228. |
@lgoettgens Thanks. |
a5ca60c
to
91c345c
Compare
@fingolfin I think now it remains to decide where we really want recursion tracking. |
src/julia_to_gap.jl
Outdated
# Methods for those types that want `false` have to be installed; | ||
# the `GapObj` methods arising from `GAP.@install` calls are unary, | ||
# therefore the macro automatically installs such a | ||
# `_needs_tracking_julia_to_gap` method. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Methods for those types that want `false` have to be installed; | |
# the `GapObj` methods arising from `GAP.@install` calls are unary, | |
# therefore the macro automatically installs such a | |
# `_needs_tracking_julia_to_gap` method. | |
# Methods for those types that want `false` have to be installed explicitly. | |
# The `GapObj` methods arising from `GAP.@install` don't handle recursion | |
# and the macro automatically installs such a `_needs_tracking_julia_to_gap` | |
# method. |
Logically, GAP.@install
could handle conversion tracking (and thus require _needs_tracking_julia_to_gap
to return true
), we just chose not to (and we still could allow for it by providing a second such macro or an optional argument to that macro).
src/julia_to_gap.jl
Outdated
function GapObj_internal(x::Integer, cache::GapCacheDict, ::Val{recursive}) where recursive | ||
# if it fits into a GAP immediate integer, convert x to Int64 | ||
x in -1<<60:(1<<60-1) && return Int64(x) | ||
# for the general case, fall back to BigInt | ||
return julia_to_gap(BigInt(x)) | ||
return GapObj_internal(BigInt(x), cache, Val(recursive)) | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is OK to do this, but since we don't track conversion for any concrete integer types, I think it would be fine just say
function GapObj_internal(x::Integer, cache::GapCacheDict, ::Val{recursive}) where recursive | |
# if it fits into a GAP immediate integer, convert x to Int64 | |
x in -1<<60:(1<<60-1) && return Int64(x) | |
# for the general case, fall back to BigInt | |
return julia_to_gap(BigInt(x)) | |
return GapObj_internal(BigInt(x), cache, Val(recursive)) | |
end | |
@install GapObj(x::Integer) = x in -1<<60:(1<<60-1) ? Int64(x) : GapObj(BigInt(x)) |
8d3901e
to
6cc82dd
Compare
src/julia_to_gap.jl
Outdated
|
||
rec = recursive && _needs_tracking_julia_to_gap(T) | ||
if rec && recursion_dict === nothing | ||
recursion_dict = RecDict() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
recursion_dict = RecDict() | |
recursion_dict = RecDict() |
(Formatting still seems to mix 2-space and 3-space wide indents -- this line was 6 spaces intended, now it is 5?)
src/julia_to_gap.jl
Outdated
) where {S} where {T<:Union{Symbol,AbstractString}} | ||
recursion_dict::GapCacheDict, | ||
::Val{recursive}, | ||
) where {S} where {T<:Union{Symbol,AbstractString}} where {recursive} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
) where {S} where {T<:Union{Symbol,AbstractString}} where {recursive} | |
) where {S, T<:Union{Symbol,AbstractString}, recursive} |
If GAP.jl is loaded by another package then `GapObj` need not be available in the `Main` module.
- add `_needs_tracking_julia_to_gap(T)`, with default value `true` - change `GAP.@install` to install a `_needs_tracking_julia_to_gap` method - change the tracking logic: - Do not *create* a dictionary for tracking identical objects only if `_needs_tracking_julia_to_gap` for the subobject type returns `false`. - Let each object decide whether it is searched in the dictionary (in the beginning of its conversion method) or whether it gets added to the dictionary (as soon as the return value is available). For the current conversion methods for integers, rationals, etc., this means that these objects are *not* added to the dictionary, that is, identical large Julia integers are converted to nonidentical large integers in GAP; if we want to change this, we have to change their `GapObj` methods. - add a method for converting `Set{T}`: This had been added to Oscar.jl but should better be in GAP.jl. (There are tests for the return value whether the result is regarded as a set by GAP; perhaps we should restrict the allowed values of `T`.) - changed `GAP.Wrappers.Add(x::GapObj, y::GapObj, z::Int)` to `GAP.Wrappers.Add(x::GapObj, y::Any, z::Int)`, in order to avoid the automatic conversion `GapObj(y)` (Once the relevant code in Oscar.jl is changed to use `GAP.@install`, the current changes in GAP.jl do not need further changes in Oscar.jl.)
and make the assumption that `recursion_dict !== nothing` implies `recursive` explicit
The `_needs_tracking_julia_to_gap` method must be installed not only for the given type but also for its subtypes.
and add a test
- add helper functions for the recursive conversions (two are needed because some method specific variables must be created between the two parts) - do not distinguish between `nothing` and `RecDict` in the subobject conversion calls
... since this function got removed
1ab5ec2
to
d7ee6f9
Compare
The required test |
If there is a corresponding branch in the Oscar repository (or your fork) then the OscarCI will prefer that branch and the job will contain the branch name in the label (to make it clear which branch is being tested):
Unfortunately there is no way to make the branch rules take that into account, they also just look at the label. |
@benlorenz thanks for your comment. I had thought it would be a good idea to have corresponding branches with the same name in GAP.jl and Oscar.jl, And for the next breaking change in GAP.jl, should I better choose a different name for the corresponding branch in Oscar.jl? |
It is meant as a feature, to be able to test whether the new code here would work together with the code in a corresponding branch for Oscar. That job is still running but it looks promising. Since this is marked |
100% agree with @benlorenz |
Weird failure in the OscarCI test on Julia 1.10 (in Aqua tests?). What is this dependency "jl_Te1ctwpcZO [9e52b259-76e2-4c08-b46d-33fd3af550ab]" ? Memory corruption?
In a previous run the error was a bit different:
|
That is a temporary environment for the Aqua tests and there is an issue about Aqua not respecting the current environment (JuliaTesting/Aqua.jl#292) which might cause this test-environment to use a different GAP version... |
This would explain the error message
The macro |
Then let's move forward with this and make a release today |
Thanks for finally merging this pull request. |
This is a first attempt to issue #1025.
The ideas are the same as in pull request #777, which deals with "the other direction", that is, the conversion from GAP to Julia:
julia_to_gap
methods with three arguments back to ajulia_to_gap
with only one argument.GapObj
method for some Oscar objects does not work as expected for vectors of these objects, or that changes in GAP.jl break the intended behaviour ofGapObj
methods in Oscar.jl (seeGapObj
vs.julia_to_gap
#1025).The conversion from Julia to GAP is simpler than that from GAP to Julia for example because we need not care about caching different Julia target types for the same GAP object. On the other hand, the direction from Julia to GAP shall cover also the constructor
GapObj
in situations where one wants to fetch an existing GAP object stored in a Julia object.The current changes mean that all Julia to GAP conversions have to be installed via
GapObj_internal
methods that have three arguments.Then
GapObj
needs only default methods that delegate toGapObj_internal
.If we proceed like this then these changes are breaking.
For Oscar.jl, they mean that the
julia_to_gap
andGapObj
methods installed there have to be rewritten.(Currently the master branch of Oscar.jl has 23
julia_to_gap
methods, half of them insrc/GAP/oscar_to_gap
.)Here are the technical details.
introduce
GapObj_internal
, which takes always three arguments: the Julia object to be converted, the auxiliary cache object, and a Boolean indicating whether recursive conversion is wantedturn all
julia_to_gap
methods intoGapObj_internal
methodsdefine default
GapObj
methods (with 1 to 3 arguments) that callGapObj_internal
admit
nothing
as an allowed value for the auxiliary cache object, and create dictionary objects only inside the methods in question where they are needed; introduce the typeGapCacheDict
for thatprovide a macro for the installation of unary
GapObj
methods (i.e., methods that do not require the recursive tracking of subobjects w.r.t. object identity) via the syntaxGAP.@install GapObj(x::T) = f(x)