Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Toolbox packaging #285

Merged
merged 18 commits into from
Oct 16, 2020
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
b016561
Fix R2016b and add toolbox packaging.
guzman-raphael Sep 19, 2020
058227c
Move compareVersions to its maintained toolbox, rename setupDJ to +dj…
guzman-raphael Sep 22, 2020
089051f
Update tests and add compiled toolbox.
guzman-raphael Sep 22, 2020
cbe5502
Merge pull request #284 from guzman-raphael/package-main
guzman-raphael Sep 22, 2020
66d4ce8
Create Documents folder as MATLAB will not ccreate it for userpath to…
guzman-raphael Sep 23, 2020
54bcf82
Merge pull request #286 from guzman-raphael/package-main
guzman-raphael Sep 23, 2020
0d2d26b
Use version-pinned MySQL8 (known issue).
guzman-raphael Sep 23, 2020
521d773
Merge pull request #287 from guzman-raphael/package-main
guzman-raphael Sep 23, 2020
90ea78e
Update readme.
guzman-raphael Sep 23, 2020
c5bf8b4
Merge pull request #288 from guzman-raphael/package-main
guzman-raphael Sep 23, 2020
621cf3a
Add FileExchange badge.
guzman-raphael Sep 23, 2020
1772969
Merge pull request #289 from guzman-raphael/package-main
guzman-raphael Sep 23, 2020
2c263f0
Pull into prompts GHToolbox dependency handling (via GitHub).
guzman-raphael Oct 6, 2020
f6a37af
Merge pull request #290 from guzman-raphael/package-main
guzman-raphael Oct 6, 2020
09b6f6b
Update mym pointer to prod, update version, update changelog.
guzman-raphael Oct 15, 2020
ecf487d
Merge pull request #294 from guzman-raphael/package-main
guzman-raphael Oct 15, 2020
66cb341
Remove toolbox binary from tracking.
guzman-raphael Oct 16, 2020
f6c6888
Merge pull request #295 from guzman-raphael/package-main
guzman-raphael Oct 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 0 additions & 100 deletions +dj/+lib/compareVersions.m

This file was deleted.

11 changes: 1 addition & 10 deletions +dj/Connection.m
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,7 @@
% specify the connection to the database.
% initQuery is the SQL query to be executed at the start
% of each new session.
setupDJ(true);
try
mymVersion = mym('version');
assert(mymVersion.major > 2 || mymVersion.major==2 && mymVersion.minor>=6)
catch
error 'Outdated version of mYm. Please upgrade to version 2.6 or later'
end
if verLessThan('matlab', '8.6')
error 'MATLAB version 8.6 (R2015b) or greater is required'
end
dj.setup('prompt', ~dj.set('suppressPrompt'));
self.host = host;
self.user = username;
self.password = password;
Expand Down
1 change: 0 additions & 1 deletion +dj/conn.m
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@
end
end
else
% invoke setupDJ
% optional environment variables specifying the connection.
env = struct(...
'host', 'DJ_HOST', ...
Expand Down
80 changes: 80 additions & 0 deletions +dj/setup.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
function setup(varargin)
ixcat marked this conversation as resolved.
Show resolved Hide resolved
p = inputParser;
addOptional(p, 'force', false);
addOptional(p, 'prompt', true);
parse(p, varargin{:});
force = p.Results.force;
prompt = p.Results.prompt;
persistent INVOKED
if ~isempty(INVOKED) && ~force
return
end
% check MATLAB
if verLessThan('matlab', '9.1')
error('DataJoint:System:UnsupportedMatlabVersion', ...
'MATLAB version 9.1 (R2016b) or greater is required');
end
% require certain toolboxes
requiredToolboxes = {...
struct(...
'Name', 'GHToolbox', ...
'ResolveTarget', 'datajoint/GHToolbox'...
), ...
struct(...
'Name', 'mym', ...
'ResolveTarget', 'guzman-raphael/mym', ...
'Version', '2.7.3'...
)...
};
try
ghtb.require(requiredToolboxes, 'prompt', prompt);
catch ME
installPromptMsg = {
'Toolbox ''%s'' did not meet the minimum requirements.'
'Would you like to proceed with an upgrade?'
};
if strcmp(ME.identifier, 'MATLAB:undefinedVarOrClass') && (~prompt || strcmpi('yes',...
dj.internal.ask(sprintf(sprintf('%s\n', installPromptMsg{:}), 'GHToolbox'))))
% fetch
tmp_toolbox = [tempname '.mltbx'];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

notice we have the .mltbx in the commit - is this intended?
if so, how would release building work w/r/t commit sequence/mltbx, etc?
(apologies if discussed already)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was intentional to commit the compiled DataJoint.mltbx file. Have a look at here. Once we modify our FileExchange DataJoint entry it will automatically release from our linked releases. All that we will need to do is 'manually' (though this can be automated) attach the DataJoint.mltbx to the release.

Copy link
Contributor

@ixcat ixcat Oct 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that GH 'release' artefacts can be tracked separately from code itself, the GH doc link from the matlab page you referenced ( https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/about-releases ) describes a variety of ways to use GH releases. Committing binaries to the main repo does not seem to be the only way to do this, perhaps I'm mistaken. Probably better for verbal discussion at this point.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right it is a fair point. The challenge I've been trying to solve here is 'how do we guarantee that the attached binary to the release represents the one built from the source?'. Did not want maintainers to be compiling, uploading this themselves to prevent human error from releasing an incorrect version. My simple solution was to commit it as well therefore anyone needing to make a release would simply use the committed binary for this (As it is strictly used for testing). Obviously a better approach would be with some automation that compiles it for us when triggering a release but well.... we just don't have that yet. 😅

Copy link
Contributor

@ixcat ixcat Oct 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did not want maintainers to be compiling, uploading this themselves to prevent human error from releasing an incorrect version - is exactly what is happening here, is it not?

As a stopgap makes sense (not entirely convinced, esp. since it will bloat repo size footprint), but also wasn't sure if that was intended as 'stopgap' or 'long term solution', so have less reservations if it's intended as a stopgap.

anyway, point raised, think conversation is probably best in person/dev meeting from here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be blunt: it is a source code repository, not a 'bag of stuff' repository; I can see the point in blurring this line slightly for the release/ease of fetching case you mention (why i'm proposing git-lfs as an alternative instead of outright removal+ a separate less-strictly-linked tracking process), but long run having every git checkout copy contain a full download of every binary copy ever released is a waste of time/space.

git-lfs allows precise, intentional version tracking as you outline (which I agree is a good thing) without bloating checkout footprint nearly as much - you get the source code history (which is on the order of kb/low mb) and the related copy of the binary (so checkout footprint of release artefacts is is on the order of 1*{release} or worst case 1*{number of active local branches having different release archives} footprint, instead of {all releases})

r.e. 100MB - this is 'separation by file size'; going back to 'bag of stuff' point, i propose 'separation by file-role'. policy decision is up to users.

will admit for dj matlab specifically, since it is only a source zip, the .mltbx is fairly small so it might not be worth the extremely limited extra effort of using lfs; but if there is a better way anyway, i think we should use this, and am putting forward the notion that git-lfs is a better way.

the "new release not requiring building/compiling but simply attach" point (which I also agree is a good/simple approach, with the additional caveat of 'so long as the storage/bloat concern is addressed') is also supported by using a git-lfs workflow.

r.e. mym: i view this as bad practice overall, so using it as a precedent to justify in my opinion is flawed; if lfs (or something else) works better for our use case, i propose we should transition mym as well

ultimately I'm not taking a hard line here, but if we are going to discuss this, the time is now, so wanting to make sure full discussion is had.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also note for full disclosure that binaries-in-archives is kind of a pet-peeve of mine, so that may impact feedback

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ixcat Those are fair points really and I do appreciate you taking the time to articulate this. In case I had not made it clear, I had not imaged the approach outlined here would be permanent in any way. Merely meant to adopt a 'short-term' approach for this while we take some time to come up with a better approach (so that users don't have to experience delays on account to this). Ideally, we should have proper CI/CD with micro-releases such that new PR's compile binaries and test based on them. Then upon merge, binaries should be recompiled from source, automatically cut to a release, and compiled binaries attached to said release. I believe that is the typical pattern other open-source projects utilize. I'd file this under 'dev process improvements' that we need to prioritize for (including migrating to GitHub Actions and utilizing the 'secret encryption' method described previously that would do away with the stage hassles).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually now that I have visited this through a bit more, I do also favor not including mltbx directly within the commit, but rather attach it to the releases. I must agree that it is not perfect as it does leave a room for some human error as pointed about by @guzman-raphael, but ultimately this is of same procedure as found for Python version releases. So I'd think best would be to gear towards finding an automation for this step in the future, but proceed on with an explicit and separate release of mltbx.

Copy link
Collaborator Author

@guzman-raphael guzman-raphael Oct 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eywalker @ixcat Thanks for the feedback, guys. Based on the discussion we had earlier on this, I have gone ahead and removed the DataJoint.mltbx binary deferring proper handling of this until we have some CD mechanism in place. Have also updated the tests to trigger a toolbox compilation.

websave(tmp_toolbox, ['https://github.com/' requiredToolboxes{1}.ResolveTarget ...
'/releases/download/' ...
subsref(webread(['https://api.github.com/repos/' ...
requiredToolboxes{1}.ResolveTarget ...
'/releases/latest'], ...
weboptions('Timeout', 60)), ...
substruct('.', 'tag_name')) ...
'/GHToolbox.mltbx'], weboptions('Timeout', 60));
% install
try
matlab.addons.install(tmp_toolbox, 'overwrite');
catch ME
if strcmp(ME.identifier, 'MATLAB:undefinedVarOrClass')
matlab.addons.toolbox.installToolbox(tmp_toolbox);
else
rethrow(ME);
end
end
% remove temp toolbox file
delete(tmp_toolbox);
% retrigger dependency validation
ghtb.require(requiredToolboxes, 'prompt', prompt);
elseif strcmp(ME.identifier, 'MATLAB:undefinedVarOrClass')
GHToolboxMsg = {
'Toolbox ''GHToolbox'' did not meet the minimum requirements.'
'Please proceed to install it.'
};
error('DataJoint:verifyGHToolbox:Failed', ...
sprintf('%s\n', GHToolboxMsg{:}));
else
rethrow(ME)
end
end
% check mym
mymVersion = mym('version');
assert(mymVersion.major > 2 || mymVersion.major==2 && mymVersion.minor>=6, ...
'DataJoint:System:mYmIncompatible', ...
'Outdated version of mYm. Please upgrade to version 2.6 or later');
% set cache
INVOKED = true;
end
2 changes: 1 addition & 1 deletion +dj/version.m
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
function varargout = version
% report DataJoint version

v = struct('major',3,'minor',3,'bugfix',1);
v = struct('major',3,'minor',3,'bugfix',3);

if nargout
varargout{1}=v;
Expand Down
100 changes: 0 additions & 100 deletions +tests/+lib/compareVersions.m

This file was deleted.

5 changes: 0 additions & 5 deletions +tests/Main.m

This file was deleted.

4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
*.m~
mym/
*.mltbx
*.env
notebook
*getSchema.m
docker-compose.yml
.vscode
matlab.prf
win.*
macos.*
macos.*
package.prj
Empty file removed .gitmodules
Empty file.
6 changes: 5 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
- <<: *slim
env:
- MATLAB_VERSION: R2019a
- MYSQL_TAG: 8.0
- MYSQL_TAG: 8.0.18
- <<: *slim
env:
- MATLAB_VERSION: R2019a
Expand All @@ -36,4 +36,8 @@ jobs:
- <<: *slim
env:
- MATLAB_VERSION: R2018b
- MYSQL_TAG: 5.7
- <<: *slim
env:
- MATLAB_VERSION: R2016b
- MYSQL_TAG: 5.7
Binary file added DataJoint.mltbx
Binary file not shown.
Loading