LUCK: the LUcky Compiling Kit for pdb-debuggable builds.

Overview:

Motivation

Makefile is concise and robust, but can be hard to learn for non-experienced bash user, due to its many unique operators. Since python is a much wider spread language than Makefile (need ref), porting Makefile syntax to Python would open up access to make-powered reproducibility to these python-only users, without having to learn the syntax.

There are several dimensions to score a build system. A detailed comparison is attached further below

Feature

Makefile-like syntax and laziness
Supports arbitrary python3.7 statements
Transparent source code with minimal presets defined as importable classes.
Simple DAG plotting luck graph
[TBC] Optional Static DAG specification instead of Runtime DAG construction
[TBC] integration with pytest?

Close-related alternatives:

SCons
luigi
gnu-make
spiper (predecessor)

CLI Usage `luckmake`:

usage: luckmake [-h] [-C DIRECTORY] [--abs-target ABS_TARGET] [--pdb]
              [--debug-class DEBUG_CLASS] [-V]
              target

positional arguments:
  target                the target within the namespace

optional arguments:
  -h, --help            show this help message and exit
  -C DIRECTORY, --directory DIRECTORY
                        Change to DIRECTORY before doing anything.
  --abs-target ABS_TARGET
                        [NotImplemented][TBC] the full url to the target
  --pdb                 run post-mortem pdb
  --debug-class DEBUG_CLASS
                        DEBUG_CLASS format: <CLASS_NAME:str>:<DEBUG_LEVEL:int>
                        possible values for CLASS_NAME:{{BaseRule}}.set
                        class.debug = DEBUG_LEVEL before execution example
                        --debug-class BaseRule:1
  -V, --version         print version

Documentation [TBC]

Install

install binary from github release (example for 0.0.8)

TAG=0.0.8
wget https://github.com/shouldsee/luckmake/releases/download/${TAG}/luck
wget https://github.com/shouldsee/luckmake/releases/download/${TAG}/luckmake
sudo install -m 755 luck luckmake /usr/local/bin

install binary from github tarball

requires A linux machine compatible with the binary

TAG=0.0.8
TAG=master  # for development branch
curl -sL https://github.com/shouldsee/luckmake/archive/${TAG}.tar.gz -o luckmake-${TAG}.tar.gz
tar -xvzf luckmake-${TAG}.tar.gz
cd luckmake-${TAG}/
sudo install -m 755 bin/luck* /usr/local/bin

install python scripts (pyluckmake)

requires Python >= 3.7

python3.7 -m pip install luck@https://github.com/shouldsee/luckmake/tarball/master && pyluckmake --help

Example

LUCKFILE.py for this project:

You can even use luckmake to install it to your file-system. Once changed into the project directory the bin directory is built, use bin/luckmake install or bin/luckmake build.

PREFIX = '~/.local'
DESTDIR= ""

from luck.shorts import RNS,DNS,LSC,TSSR
from luck.types import NoCacheRule as NCR

RULE=TSSR
ns = RNS()


### always use NCR for aliasing
NCR.MWF(ns, 'all', 'build install')  
### alias
NCR.MWF(ns, 'build','./bin/luckmake ./bin/luck')

### use LSC for bash command, f-string for string-completion
NCR.MWF(ns, 'install','build',
	''' 
	install -d {DESTDIR}{PREFIX}/bin/
	install -m 755 ./bin/luckmake {DESTDIR}{PREFIX}/bin/
	install -m 755 ./bin/luck {DESTDIR}{PREFIX}/bin/
	''')

RULE.MWF(ns, './bin/luckmake ./bin/luck', 'luck/types.py', 
	'''
	python3.7 -m PyInstaller cli.spec --distpath ./bin --clean
	### this command would produce ./bin/luckmake and ./bin/luck
	''')

### specify external root nodes with RULE=TSSR. 
RULE.MWF(ns, 'luck/types.py', None)
RULE.MWF(ns, 'build.sh',None)


### use NCR for commands that should always execute
NCR.MWF(ns, 'error',  '', 'echo 1231243231 && false')
NCR.MWF(ns, 'pybuild','', 'python3.7 -m pip install . --user && pytest . && rm bin -rf')

TSSR.MWF(ns,'example-ece264-hw04.dir',None)

NCR.MWF(ns, 'test.sh', '', '''
cd example-ece264-hw04.dir/
python3.7 README-example.py build ./hw04
''')
NCR.MWF(ns, 'count-line', '', 'wc example-ece264-hw04.dir/{*E.py,Makefile} -c')
NCR.MWF(ns, 'clean', '','''
	rm -rf bin/* build/*
	''')

charcount

1553 example-ece264-hw04.dir/LUCKFILE.py
2638 example-ece264-hw04.dir/v1.LUCKFILE.py
 895 example-ece264-hw04.dir/Makefile

Sciprting Syntax and Tips

Tips

importing module

sys.path in LUCKFILE.py will be provided by the luckmake binary and not the system python installation. If you want to import numpy, best to place a sys.path.append("PATH_TO_NUMPY_PARENT_DIR") before import numpy
get the PATH_TO_NUMPY_PARENT_DIR with python3.7 -c "import numpy as mod; print(mod.__file__)"
if the output is "/home/shouldsee/.local/lib/python3.7/site-packages/numpy/init.py", then you should sys.append("/home/shouldsee/.local/lib/python3.7/site-packages/")

Cache a rule or not?

NoCachedRule (NCR) vs TimeSizeStampRule (TSSR):
it's important to select NCR / TSSR carefully to avoid later confusion.
use NCR when the output cannot be cached. This is true for any command-like rule
use TSSR or other stamped rule when the output is one or more files.

namespace utility

RNS:    RuleNameSpace
DNSUB:  DelayedNameSpaceSUBclass
LSC:    LoggedShellCommand

callable to execute during build time

pyfunc: 
AutoCmd:
MFP:    MakeFilePattern:

rule classes

BaseRule:
TimeSizeStampRule:
MD5StampRule:
NoCacheRule:

generic methods

<klass>.M(namespace, key, *init_attrs):     
	M for modify
	============
	instead of creating a new instance, assign it to a <namespace> under <key>.
	
	Paritularly, for `luck.defer.BaseRule`, the modify() would return a list of BaseRule() 
	by iterating over `key.split()`.


	Params
	========
	namespace:  a dict-like object
	key:        the name to assign to wihtin <namespace>
	init_attrs: passed to `klass(*init_attrs)`

<klass>.MWF(namespace, key, *init_attrs, frame):
	MWF for modify with frame
	==========================
	This class method is used for automatically casting a recipe in init_attrs
	into a suitable lambda expression with fstring using FstringShellCommand(),
	so that you do not have to type 
	`RULE.M(ns, 'foo', None, lambda c:LSC(f'echo {GCC}')` 
	all the time, and just type instead
	`RULE.MWF(ns, 'foo', None, 'echo {GCC}')`

Below is a side by side comparison of Makefile and LUCKFILE.py. You would notice that LUCKFILE.py is significantly more verbose and have more quotes, but there is definitely space for a more concise grammar.

Makefile is adapted from ECE264 2019 problems

while make looks for Makefile in current directory, luck looks for "LUCKFILE.py"

cd example-ece264-hw04.dir
make clean
make testall

luckmake clean
luckmake testall
echo [FIN]

More examples

Makefile

./example-ece264-hw04.dir/Makefile

WARNING = -Wall -Wshadow --pedantic -Wno-unused-variable
ERROR = -Wvla -Werror
GCC = gcc -std=c99 -g $(WARNING) $(ERROR) 

TESTFALGS = -DTEST_COUNTCHAR -DTEST_PRINTCOUNTS

SRCS = main.c filechar.c
OBJS = $(SRCS:%.c=%.o)

%.o : %.c
	$(GCC) $(TESTFALGS) -c $*.c -o $@

hw04: $(OBJS) 
	$(GCC) $(TESTFALGS) $(OBJS) -o hw04

test1: hw04
	./hw04 inputs/2016 > output16
	diff output16 expected/

LUCKFILE.py syntax M: Use `modify()` statements

./example-ece264-hw04.dir/LUCKFILE_syntax_M.py

from luck.shorts import RNS,DNS,ACMD,MFP,LSC
from luck.types import TimeSizeStampRule as RULE
from luck.types import NoCacheRule

ns = RNS.subclass('MainRNS')() 
patterns = DNS.subclass('PatternNS')()

WARNING = "-Wall -Wshadow --pedantic -Wno-unused-variable"
ERROR = "-Wvla -Werror"
TESTFALGS = "-DTEST_COUNTCHAR -DTEST_PRINTCOUNTS"

GCC = f"gcc -std=c99 -g {WARNING} {ERROR} {TESTFALGS}"

SRCS = "main.c filechar.c"
OBJS = ' '.join([x[:-2]+'.o' for x in SRCS.split()]) ## pure python func!


RULE.M(ns, OBJS, SRCS,  ACMD(patterns))
RULE.M(ns, SRCS, None,  None)


MFP.M(patterns,
	0, '%.o','%.c', 
	lambda c: LSC(f'{GCC} {TESTFALGS} -c {c.i[0]} -o {c.o[0]}'))


RULE.M(ns, './hw04', OBJS, lambda c:LSC(f'''
	{GCC} {TESTFALGS} {OBJS} -o {c.o[0]}
	'''))


RULE.M(ns, 'test1',  './hw04',  lambda c:LSC(f'''
	{c.i[0]} inputs/2016 > output16
	diff output16 expected/expected16
	echo [passed] test1
	'''))


if __name__ == '__main__':
	from luck.cli import luck_main
	luck_main(ns)

LUCKFILE.py syntax A: Use assign statements

./example-ece264-hw04.dir/LUCKFILE_syntax_A.py

from luck.types import *
## create RuleNameSpace and set default
ns = RNS.subclass('MainRNS')(ruleFactory=TimeSizeStampRule) 
## create patterns namespace 
patterns = DNSUB('PatternNS')

WARNING = "-Wall -Wshadow --pedantic -Wno-unused-variable"
ERROR = "-Wvla -Werror"
TESTFALGS = "-DTEST_COUNTCHAR -DTEST_PRINTCOUNTS"

GCC = f"gcc -std=c99 -g {WARNING} {ERROR} {TESTFALGS}"

SRCS = "main.c filechar.c"
OBJS = ' '.join([x[:-2]+'.o' for x in SRCS.split()]) ## pure python func!

ns[OBJS] = (SRCS, AutoCmd(patterns))
ns[SRCS] = (None,None, TimeSizeStampRule)

patterns[0] = MakefilePattern(
	'%.o','%.c', 
	lambda x: LSC(f'{GCC} {TESTFALGS} -c {x.inputs[0]} -o {x.outputs[0]}'))

ns['./hw04'] = (f'{OBJS}',
	lambda c:LSC(f'''
		{GCC} {TESTFALGS} {OBJS} -o {c.o[0]}
		'''))
### c.i, c.o instead of $< $@
ns['test1'] = ('./hw04', lambda c:LSC(f'''
	{c.i[0]} inputs/2016 > output16
	diff output16 expected/expected16
	echo [passed] test1
	'''))

if __name__ == '__main__':
	from luck.cli import luck_main
	luck_main(ns)

./example-ece264-hw04.dir/v1.LUCKFILE.py Older version

from luck.types import ExternalFileTask, LinkedTask, TStampedLocalTarget, LoggedShellCommand, rstrip
# or just import *

WARNING = "-Wall -Wshadow --pedantic -Wno-unused-variable"
ERROR = "-Wvla -Werror"
TESTFALGS = "-DTEST_COUNTCHAR -DTEST_PRINTCOUNTS"

GCC = "gcc -std=c99 -g {WARNING} {ERROR} {TESTFALGS}".format(**locals())

SRCS = "main.c filechar.c".split()
OBJS = [rstrip(x,'.c')+'.o' for x in SRCS]


class hw04(LinkedTask):
	requires = lambda self: [ExternalFileTask('main.c'),ExternalFileTask('filechar.c')]
	output   = lambda self: TStampedLocalTarget('hw04')
	def run(self): 
		LoggedShellCommand([GCC, [x.path for x in self.input()], '-o', self.output().path]); 
		super().run()


class test1(LinkedTask):
	requires = lambda self: [hw04()]
	output = lambda self: TStampedLocalTarget('output2016.passed')
	def run(self):
		LoggedShellCommand(['./'+self.input()[0].path, 'inputs/2016 > output2016','&&','diff','expected/expected16','output2016'])		
		super().run()

Improvements:

[design] How to build in a different directory?
[cli] add runtime variable '-e' overriding luckmake install PREFIX=blah
[cli] add -B --always-make
[design] introduce hierarchical calls RULE.M(ns, output, input=RULE.M(ns, output, input)). This would allow easy specification of upstream.
[todo] better LSC with stdout stderr with os.system does not return stdout...
[todo] add tests for "luckmake" instead of "pyluckmake"
[urg,perf] multi-worker build.. preferably a portable implementation
[sug] adding utility function for gdb upon exception
[sug] profiling gprof
[sug,ada] dry run dependency graph
[doc] Add docs/
[doc] automate README.md generation.
[port] Python is not the best language for writing a build system because of its poor portability. I am using python because it is more expressive than a static yaml/json file. It would be great if we can write a parser in c/cpp/go to emulate a reduced version of python.

Changelog:

0.0.8
- [added] luck graph to plot dependency graph
- [dep] "graphviz" as a dependency for luck-graph
- [change] refactor subcommanding in luck/cli.py
- [added] luck.defer.RuleNameSpace._module_file
- [change] luck.shell.LoggedShellCommand to use subprocess.run instead subprocess.call
0.0.6
- [done] rename "luckbd" to "luckmake"
- [added] glob matching for luck/*.py, luck/**.py at defer.str_expand()
- [added] FstringShellCommand() with {BaseRule,MakefilePattern}.modifyWithFrame() MWF() for shorthand This is to provide auto type casting from str to lambda within the calling context
  - delaying f-string expression is notoriously difficult. see SO#42497625. Current workaround passes frame pointer in klass.MWF(*args,frame), but an additional string prefix like lf(c,*args)"echo {args[0]}" is desirable for debuggable build.
- [done] enabling luck/* with glob.
- [done] enabling luck/** to match all file in directory.
  - implemented in str_expand()
  - excluding _luck __pycache__ by 0.0.6 . need to add [suffix matching, blacklist configuration]
0.0.5
- 'fileA fileB' would share a ident_same() method [TBC] adapt the logic for AutoCmd() dispatching
- avoid recomputing the same rule that would output mutiple files
- rework defer.BaseRule.{__init__,check,build}
- rework rule_stamp.StampRule.{ident_same,build_after}
- added --debug-class <CLASS_NAME>:<DEBUG_LEVEL>
0.0.4
- provide "luck", "luckbd" in binary, built with pyinstaller
- provide "pyluck", "pyluck" for python3.7
- rename "luck-build" to "pyluckbd"
- stripped heading # -*- coding: future_fstrings -*-
- now requires python3.7 for f-strings
0.0.3
- added luck-build --pdb
- added modifier syntax and .M methods
- added luck.shorts for shortcuts, and remove all shortcuts from luck.types.
not required in new syntaxes since 0.0.2
- ~~[urg,urg] shall I use f-string?~~ now depends on future-fstrings
  and heading # -*- coding: future_fstrings -*-
- ~~[sug,urg] get rid of super().run() for subclasses of LinkedTask~~
- [sug] use "self.input().path" or "self.input()" ?
~~[urgent] NoCacheRule() and TimeStampRule()~~ - TSR would cache status for non-existing files. event for TSR.M(ns, 'all', 'build'). However this is only a symbloic inheritance, and the built-status of build should resolve to its upstream and not the pseudofile.

Current workaround is to use NCR.M instead of TSR.M for symbolic tasks, but NCR will always be rebuilt regarless of its upstream.

Solution A: add a rule class to achieve this behaviour~~

Detailed comparison

learning cost: subjective and scenario-dependent
- Makefile: easier for a bash user. less tutorial
- LUCKFILE.py: no tutorial yet, only example available. easy for a python user
coarseness: high-level or low-level
- Makefile: dag level
- LUCKFILE.py: dag
interface: cli? gui? web? diagnostic tools? easy to debug?
- Makefile: many tools exist for static analysis. gmd for debugging (yet to try)
- LUCKFILE.py: simple cli only for now. pdb for debugging import pdb;pdb.set_trace()
speed: for constructing dag and execution
- Makefile: should be faster as is clang, multiprocessed
- LUCKFILE.py: single-process for now.
persistence: whether result is saved to disk, and how easy to enter a corrupted state
- Makefile: save mtime for persistence
- LUCKFILE.py: use mtime+size or md5sum
portability: easy to install? backend for different platform?
- Makefile: sudo apt install make
- LUCKFILE.py: pip install luck@https://github.com/shouldsee/luck/tarball/master0
extensibility: how easy to write a plugin?
- Makefile: possible embedded Scheme (.scm ) (need to learn LISP aside from Makefile)
- LUCKFILE.py: write python class/callable to inject dependency, same lang as LUCKFILE.py
relative or absolute path:
- Makefile: NA
- LUCKFILE.py: NA

Alternatives and Refs

https://medium.com/@mattia512maldini/looking-for-a-makefile-alternative-6e7f795b5cad
https://alternativeto.net/software/gnu-make/
{name}: {stars}k, {comments}
go / python
go-task: 1.7k go (liked)
luigi: 13k, python (included, no DSL)
joblib: 2k, python (persistent cache)
scons: 0.8k, python (DSL, high-level)
snakemake: 0.5k, python (similar, but use DSL)
galaxy: 0.7k python (use-xml)
waf: 0.078k, python
sake: 0.3k, python
redo: 1.4k, python (use-bash)
java / groovy
gradle: 10.5k, groovy maybe? see bazel
maven: 2.1k, na, xml-based
buck: 7.3k, java android-spec
c / cpp
cmake: na, cpp?
gnumake: na, no
premake: 1.6K, clang
makeme: 0.02k, clang
ninja build: 5.5k, c or cpp
other
ant: na, xml
rake: na, ruby, maybe
sbt: na, java/scala, maybe
not-compared
apache-airflow: too big

Name		Name	Last commit message	Last commit date
Latest commit History 33 Commits
.pybuild/pythonX.Y_3.5		.pybuild/pythonX.Y_3.5
_luck		_luck
bin		bin
example-ece264-hw04.dir		example-ece264-hw04.dir
example-htslib-small.dir		example-htslib-small.dir
examples		examples
luck		luck
test/pointer		test/pointer
.gitignore		.gitignore
.travis.yml		.travis.yml
LUCKFILE.py		LUCKFILE.py
LUCKFILE.py.dot		LUCKFILE.py.dot
LUCKFILE.py.dot.svg		LUCKFILE.py.dot.svg
Makefile		Makefile
README.md		README.md
cli.spec		cli.spec
requirements.txt		requirements.txt
setup.py		setup.py
test_all.py		test_all.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

LUCK: the LUcky Compiling Kit for pdb-debuggable builds.

Overview:

Motivation

Feature

Close-related alternatives:

CLI Usage `luckmake`:

Documentation [TBC]

Install

install binary from github release (example for 0.0.8)

install binary from github tarball

install python scripts (pyluckmake)

Example

LUCKFILE.py for this project:

charcount

Sciprting Syntax and Tips

Tips

importing module

Cache a rule or not?

namespace utility

callable to execute during build time

rule classes

generic methods

More examples

Makefile

LUCKFILE.py syntax M: Use `modify()` statements

LUCKFILE.py syntax A: Use assign statements

Improvements:

Changelog:

Detailed comparison

Alternatives and Refs

Example Dependency Graph

About

Releases 6

Packages

Languages

shouldsee/luckmake

Folders and files

Latest commit

History

Repository files navigation

LUCK: the LUcky Compiling Kit for pdb-debuggable builds.

Overview:

Motivation

Feature

Close-related alternatives:

CLI Usage luckmake:

Documentation [TBC]

Install

install binary from github release (example for 0.0.8)

install binary from github tarball

install python scripts (pyluckmake)

Example

LUCKFILE.py for this project:

charcount

Sciprting Syntax and Tips

Tips

importing module

Cache a rule or not?

namespace utility

callable to execute during build time

rule classes

generic methods

More examples

Makefile

LUCKFILE.py syntax M: Use modify() statements

LUCKFILE.py syntax A: Use assign statements

Improvements:

Changelog:

Detailed comparison

Alternatives and Refs

Example Dependency Graph

About

Resources

Stars

Watchers

Forks

Releases 6

Packages 0

Languages

CLI Usage `luckmake`:

LUCKFILE.py syntax M: Use `modify()` statements

Packages