-
Notifications
You must be signed in to change notification settings - Fork 12
CellBlender MakeFile
We're using a somewhat complicated build/ID system for CellBlender and this document offers a bit of an explanation.
Our build system is complicated because we wanted the ability to "fingerprint" the version of CellBlender that any user is running so we can tell:
- Which version they're running (which git commit)
- If they've modified any files from our release
The way we accomplish this is to maintain a list (literally a Python list) of files that are part of each release. Then we can compute a hash (currently the SHA1) of each of those files to come up with an ID of the currently running version.
The list of files is maintained in "cellblender_source_info.py". That's the one file that we should manually update when we add or remove source files from our build. It contains a pretty obvious list of files in "cellblender_source_list" within the "cellblender_info" dictionary. Here's a shortened version of what that dictionary entry looks like:
"cellblender_source_list": [
"__init__.py",
"cellblender_source_info.py",
"data_model.py",
"parameter_system.py",
"cellblender_properties.py",
"cellblender_panels.py",
"cellblender_operators.py",
"cellblender_molecules.py",
"icons"+os.sep+"cellblender_icon.png",
"icons"+os.sep+"mol_sel.png",
"SimControl.java",
"glyph_library.blend",
"io_mesh_mcell_mdl"+os.sep+"__init__.py",
"io_mesh_mcell_mdl"+os.sep+"export_mcell_mdl.py",
"io_mesh_mcell_mdl"+os.sep+"mdlmesh_parser.py",
"data_plotters"+os.sep+"java_plot"+os.sep+"PlotData.jar",
"mdl"+os.sep+"__init__.py",
"bng"+os.sep+"__init__.py",
"bng"+os.sep+"sbml2blender.py",
"bng"+os.sep+"sbml2json.py",
],
The running version of CellBlender can go through that list of files to compute and report the current version that we refer to as the "CellBlender Source ID". If the user has changed any of those files, that difference will produce a different CellBlender Source ID.
But that still doesn't tell us which git commit created any particular Source ID. To accomplish that, we simply compute the Source ID at make time, and use it to update a file named "cellblender_id.py". Here's what that entire 1-line file looks like:
cellblender_id = '0e0d7fdda03628f919d1209f1f356331a2f6b501'
Since this file is tracked by git, we can search the git database to find any given ID in the file named "cellblender_id.py". Here's how to do it:
git log -S'0e0d7fdda03628f919d1209f1f356331a2f6b501' -- cellblender_id.py
In this example, that command will show something like this:
commit c5a306cd21fbd9461e1870fc38f326b6312330ee
Author: Bob Kuczewski <email address>
Date: Thu Sep 18 15:28:09 2014 -0700
Reworked the panels to combine model initialization with run simulation.
This allows us to identify the commit used to generate any version of CellBlender that any end user might report to us ... assuming that we've run "make" before each commit. Running "make" will update the cellblender_id.py file. Failing to run "make" before a commit will result in the same "cellblender_id.py" file for more than one commit. Here's an example:
$ git log -S'101ff19d1' -- cellblender_id.py
commit 42e307be08f8952c48613004351eb215a30a9035
Date: Mon Jul 13 13:46:41 2015 -0700
commit 6f46e40ffa7dd532ee3b1c78cf00ca6a4c86fc3f
Date: Wed Jul 1 19:42:27 2015 -0700
commit 18ba34c3ceb94e752d79f4dbb03723e77f26c522
Date: Tue Jun 23 12:02:12 2015 -0700
Since all 3 of those commits were made with the same cellblender_id.py file, a git search for that ID will turn up all three commits. Which one is correct? This can be resolved by refreshing (recalculating) the CellBlender ID within Blender and comparing it to the properly calculated ID for each of those commits (typically by checking each one out, and then running "make" to calculate the ID that should have been stored with the commit).
Now on to the make file itself ...
Since we have this nice list of source files in a python list already, it makes sense to use it in the make file (rather than maintaining two separate but identical lists). We accomplish this by attaching a "main" section to the end of our "cellblender_source_info.py" file. That "main" section is called when cellblender_source_info.py is run via the python interpreter on the command line:
python cellblender_source_info.py
When run like that, it will not only regenerate the cellblender_id.py file, but will also print (to standard out) a list of the file names contained in the cellblender_source_list. This is exactly what make needs. So there's a line in the make file that does this:
SOURCES = $(shell python cellblender_source_info.py)
The $(SOURCES) variable can then be used in dependency rules and for other handy purposes (like selecting files to include in a zip file).
The moral of the story is that it's important to keep the source list (in cellblender_source_info.py) updated with whatever files we want in the final zip file. The ONLY file not included in that list is the SHA1 file "cellblender_id.py" itself because that file is auto-generated to contain the SHA1 of the files in the list. It cannot be included in the SHA1 for obvious reasons (since it IS the SHA1).
There's one more wrinkle in this system related to older versions.
We've been reporting the SHA1 of the source files to the user since April 4th, 2013. So we can identify a particular version that the user is running since that time. However, we've only been storing that SHA1 in git since June 26th, 2014. This means that we can't easily search the git database for CellBlender Source IDs earlier than that. We could probably come up with some way to automatically check out each version, compute the CellBlender Source ID, and then associate that ID with the git commit ID. Fortunately, we don't have much need for that, so it hasn't been done. There are, however, a few cases where we might want to be able to associate the Source ID reported by CellBlender with a particular commit. Those are the candidate releases that we've tagged in git before June 26th, 2014. Here's that list:
May 15th, 2013
git tag: v1.0RC
git commit: a1abdd291b75176d6581df41329781ae5d5e1b7d
CellBlender Source ID: 82a39aca4fb6c5b3fb455ee96896fcd4018dbf5c
May 19th, 2013
git tag: v1.0RC2
git commit: fd1bcc6194f5d7f4afda3c474331787d9d2cbd12
CellBlender Source ID: 4ee693eb5dccc2698950ac88554d96bde9f0d7e5
April 20th, 2014
git tag: v1.0RC3
git commit: e804f0e44b792bfa04df0c55393f6433ad9a67f1
CellBlender Source ID: a6a38d85b6a1efb362dba46b3b07f944a49948ed
(also released with ID: 7edb9d06a4a2f5a3ce195b2a0e9899d9b1e3fd34)
So if a user is running any of those versions, they should be able to report one of those Source IDs which identifies the source they're using. Any other IDs (since June 26th, 2014) can be searched directly in git:
git log -S'### Source ID Here #####' -- cellblender_id.py
Source ID's and Commit ID's of interest:
CellBlender 1.1 (used in Book Chapter):
cellblender_id = ce75cdd9c7eb5374b55dfa499bf2155f3fc45595
Git Commit = 41aff8096df24313b18bf2739b7ec8ab9aaedfda
CellBlender 1.1 with incorrect CellBlender ID
cellblender_id = 004e97d5a857308f5401077fa3c4c0adae9bf438
Git Commit = 681a9cc05724ea0c173e3876d90805647b33f4a5
This is the same code as git commit 41aff8096df24313b18bf2739b7ec8ab9aaedfda (above),
but the CellBlender ID (SHA1) in the file doesn't match the SHA1 computed from the code.
This mismatch will be reported with the following warnings in the Preferences panel:
[Warning Icon] Blend File version doesn't match CellBlender version
[Nuclear Icon] [Button: Upgrade Blend File to Current Version]
[Warning Icon] Note: Saving this file will FORCE an upgrade!!!
CellBlender 1.0.1
cellblender_id = 101ff19d19e96b9f3b8879725e7a2e1458d987ef
Git Commit = 18ba34c3ceb94e752d79f4dbb03723e77f26c522
Same CellBlender ID in Git Commit: 6f46e40ffa7dd532ee3b1c78cf00ca6a4c86fc3f
Same CellBlender ID in Git Commit: 42e307be08f8952c48613004351eb215a30a9035
CellBlender 1.0
cellblender_id = 656f0fec755f4db9a66a53b3064850cac5b6a33b
Git Commit = b24d1f06d01e8fb26a16e6770bcd8f973c76c605
Same CellBlender ID in Git Commit: ff79950fb90998d962632182ad1d2dbf53929d13
Same CellBlender ID in Git Commit: 18ba34c3ceb94e752d79f4dbb03723e77f26c522