From f42e00770915d36e7118b57cbc19531fa38ad9a2 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sat, 28 Dec 2019 23:13:59 -0600 Subject: [PATCH 01/41] Created initial implementation of revamped test.py This is the start of reworking littlefs's testing framework based on lessons learned from the initial testing framework. 1. The testing framework needs to be _flexible_. It was hacky, which by itself isn't a downside, but it wasn't _flexible_. This limited what could be done with the tests and there ended up being many workarounds just to reproduce bugs. The idea behind this revamped framework is to separate the description of tests (tests/test_dirs.toml) and the running of tests (scripts/test.py). Now, with the logic moved entirely to python, it's possible to run the test under varying environments. In addition to the "just don't assert" run, I'm also looking to run the tests in valgrind for memory checking, and an environment with simulated power-loss. The test description can also contain abstract attributes that help control how tests can be ran, such as "leaky" to identify tests where memory leaks are expected. This keeps test limitations at a minimum without limiting how the tests can be ran. 2. Multi-stage-process tests didn't really add value and limited what the testing environment. Unmounting + mounting can be done in a single process to test the same logic. It would be really difficult to make this fail only when memory is zeroed, though that can still be caught by power-resilient tests. Requiring every test to be a single process adds several options for test execution, such as using a RAM-backed block device for speed, or even running the tests on a device. 3. Added fancy assert interception. This wasn't really a requirement, but something I've been wanting to experiment with for a while. During testing, scripts/explode_asserts.py is added to the build process. This is a custom C-preprocessor that parses out assert statements and replaces them with _very_ verbose asserts that wouldn't normally be possible with just C macros. It even goes as far as to report the arguments to strcmp, since the lack of visibility here was very annoying. tests_/test_dirs.toml:186:assert: assert failed with "..", expected eq "..." assert(strcmp(info.name, "...") == 0); One downside is that simply parsing C in python is slower than the entire rest of the compilation, but fortunately this can be alleviated by parallelizing the test builds through make. Other neat bits: - All generated files are a suffix of the test description, this helps cleanup and means it's (theoretically) possible to parallelize the tests. - The generated test.c is shoved base64 into an ad-hoc Makefile, this means it doesn't force a rebuild of tests all the time. - Test parameterizing is now easier. - Hopefully this framework can be repurposed also for benchmarks in the future. --- Makefile | 7 +- emubd/lfs_emubd.c | 16 +- lfs_util.h | 8 +- scripts/explode_asserts.py | 211 ++++++++++++++++ scripts/test_.py | 494 +++++++++++++++++++++++++++++++++++++ tests_/test_dirs.toml | 209 ++++++++++++++++ 6 files changed, 932 insertions(+), 13 deletions(-) create mode 100755 scripts/explode_asserts.py create mode 100755 scripts/test_.py create mode 100644 tests_/test_dirs.toml diff --git a/Makefile b/Makefile index 95c26dac..e17b427f 100644 --- a/Makefile +++ b/Makefile @@ -59,15 +59,20 @@ test: \ test_corrupt @rm test.c test_%: tests/test_%.sh - ifdef QUIET @./$< | sed -nu '/^[-=]/p' else ./$< endif +test_: + ./scripts/test_.py $(TFLAGS) + -include $(DEP) +%?: + @echo '$($*)' + lfs: $(OBJ) $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ diff --git a/emubd/lfs_emubd.c b/emubd/lfs_emubd.c index 60c3470d..9a29ff1d 100644 --- a/emubd/lfs_emubd.c +++ b/emubd/lfs_emubd.c @@ -158,9 +158,9 @@ int lfs_emubd_read(const struct lfs_config *cfg, lfs_block_t block, uint8_t *data = buffer; // Check if read is valid - assert(off % cfg->read_size == 0); - assert(size % cfg->read_size == 0); - assert(block < cfg->block_count); + LFS_ASSERT(off % cfg->read_size == 0); + LFS_ASSERT(size % cfg->read_size == 0); + LFS_ASSERT(block < cfg->block_count); // Zero out buffer for debugging memset(data, 0, size); @@ -213,9 +213,9 @@ int lfs_emubd_prog(const struct lfs_config *cfg, lfs_block_t block, const uint8_t *data = buffer; // Check if write is valid - assert(off % cfg->prog_size == 0); - assert(size % cfg->prog_size == 0); - assert(block < cfg->block_count); + LFS_ASSERT(off % cfg->prog_size == 0); + LFS_ASSERT(size % cfg->prog_size == 0); + LFS_ASSERT(block < cfg->block_count); // Program data snprintf(emu->child, LFS_NAME_MAX, "%" PRIx32, block); @@ -228,7 +228,7 @@ int lfs_emubd_prog(const struct lfs_config *cfg, lfs_block_t block, } // Check that file was erased - assert(f); + LFS_ASSERT(f); int err = fseek(f, off, SEEK_SET); if (err) { @@ -287,7 +287,7 @@ int lfs_emubd_erase(const struct lfs_config *cfg, lfs_block_t block) { lfs_emubd_t *emu = cfg->context; // Check if erase is valid - assert(block < cfg->block_count); + LFS_ASSERT(block < cfg->block_count); // Erase the block snprintf(emu->child, LFS_NAME_MAX, "%" PRIx32, block); diff --git a/lfs_util.h b/lfs_util.h index 80fab554..a131a3f5 100644 --- a/lfs_util.h +++ b/lfs_util.h @@ -51,28 +51,28 @@ extern "C" // Logging functions #ifdef LFS_YES_TRACE #define LFS_TRACE(fmt, ...) \ - printf("lfs_trace:%d: " fmt "\n", __LINE__, __VA_ARGS__) + printf("%s:%d:trace: " fmt "\n", __FILE__, __LINE__, __VA_ARGS__) #else #define LFS_TRACE(fmt, ...) #endif #ifndef LFS_NO_DEBUG #define LFS_DEBUG(fmt, ...) \ - printf("lfs_debug:%d: " fmt "\n", __LINE__, __VA_ARGS__) + printf("%s:%d:debug: " fmt "\n", __FILE__, __LINE__, __VA_ARGS__) #else #define LFS_DEBUG(fmt, ...) #endif #ifndef LFS_NO_WARN #define LFS_WARN(fmt, ...) \ - printf("lfs_warn:%d: " fmt "\n", __LINE__, __VA_ARGS__) + printf("%s:%d:warn: " fmt "\n", __FILE__, __LINE__, __VA_ARGS__) #else #define LFS_WARN(fmt, ...) #endif #ifndef LFS_NO_ERROR #define LFS_ERROR(fmt, ...) \ - printf("lfs_error:%d: " fmt "\n", __LINE__, __VA_ARGS__) + printf("%s:%d:error: " fmt "\n", __FILE__, __LINE__, __VA_ARGS__) #else #define LFS_ERROR(fmt, ...) #endif diff --git a/scripts/explode_asserts.py b/scripts/explode_asserts.py new file mode 100755 index 00000000..4e56710e --- /dev/null +++ b/scripts/explode_asserts.py @@ -0,0 +1,211 @@ +#!/usr/bin/env python3 + +import parsy as p +import re +import io +import sys + +ASSERT_PATTERN = p.string('LFS_ASSERT') | p.string('assert') +ASSERT_CHARS = 'La' +ASSERT_TARGET = '__LFS_ASSERT_{TYPE}_{COMP}' +ASSERT_TESTS = { + 'int': """ + __typeof__({lh}) _lh = {lh}; + __typeof__({lh}) _rh = (__typeof__({lh})){rh}; + if (!(_lh {op} _rh)) {{ + printf("%s:%d:assert: " + "assert failed with %"PRIiMAX", expected {comp} %"PRIiMAX"\\n", + {file}, {line}, (intmax_t)_lh, (intmax_t)_rh); + exit(-2); + }} + """, + 'str': """ + const char *_lh = {lh}; + const char *_rh = {rh}; + if (!(strcmp(_lh, _rh) {op} 0)) {{ + printf("%s:%d:assert: " + "assert failed with \\\"%s\\\", expected {comp} \\\"%s\\\"\\n", + {file}, {line}, _lh, _rh); + exit(-2); + }} + """, + 'bool': """ + bool _lh = !!({lh}); + bool _rh = !!({rh}); + if (!(_lh {op} _rh)) {{ + printf("%s:%d:assert: " + "assert failed with %s, expected {comp} %s\\n", + {file}, {line}, _lh ? "true" : "false", _rh ? "true" : "false"); + exit(-2); + }} + """, +} + +def mkassert(lh, rh='true', type='bool', comp='eq'): + return ((ASSERT_TARGET + "({lh}, {rh}, __FILE__, __LINE__, __func__)") + .format( + type=type, TYPE=type.upper(), + comp=comp, COMP=comp.upper(), + lh=lh.strip(' '), + rh=rh.strip(' '))) + +def mkdecl(type, comp, op): + return (( + "#define "+ASSERT_TARGET+"(lh, rh, file, line, func)" + " do {{"+re.sub('\s+', ' ', ASSERT_TESTS[type])+"}} while (0)\n") + .format( + type=type, TYPE=type.upper(), + comp=comp, COMP=comp.upper(), + lh='lh', rh='rh', op=op, + file='file', line='line', func='func')) + +# add custom until combinator +def until(self, end): + return end.should_fail('should fail').then(self).many() +p.Parser.until = until + +pcomp = ( + p.string('==').tag('eq') | + p.string('!=').tag('ne') | + p.string('<=').tag('le') | + p.string('>=').tag('ge') | + p.string('<').tag('lt') | + p.string('>').tag('gt')); + +plogic = p.string('&&') | p.string('||') + +@p.generate +def pstrassert(): + yield ASSERT_PATTERN + p.regex('\s*') + p.string('(') + p.regex('\s*') + yield p.string('strcmp') + p.regex('\s*') + p.string('(') + p.regex('\s*') + lh = yield pexpr.until(p.string(',') | p.string(')') | plogic) + yield p.string(',') + p.regex('\s*') + rh = yield pexpr.until(p.string(')') | plogic) + yield p.string(')') + p.regex('\s*') + op = yield pcomp + yield p.regex('\s*') + p.string('0') + p.regex('\s*') + p.string(')') + return mkassert(''.join(lh), ''.join(rh), 'str', op[0]) + +@p.generate +def pintassert(): + yield ASSERT_PATTERN + p.regex('\s*') + p.string('(') + p.regex('\s*') + lh = yield pexpr.until(pcomp | p.string(')') | plogic) + op = yield pcomp + rh = yield pexpr.until(p.string(')') | plogic) + yield p.string(')') + return mkassert(''.join(lh), ''.join(rh), 'int', op[0]) + +@p.generate +def pboolassert(): + yield ASSERT_PATTERN + p.regex('\s*') + p.string('(') + p.regex('\s*') + expr = yield pexpr.until(p.string(')')) + yield p.string(')') + return mkassert(''.join(expr), 'true', 'bool', 'eq') + +passert = p.peek(ASSERT_PATTERN) >> (pstrassert | pintassert | pboolassert) + +@p.generate +def pcomment1(): + yield p.string('//') + s = yield p.regex('[^\\n]*') + yield p.string('\n') + return '//' + s + '\n' + +@p.generate +def pcomment2(): + yield p.string('/*') + s = yield p.regex('((?!\*/).)*') + yield p.string('*/') + return '/*' + ''.join(s) + '*/' + +@p.generate +def pcomment3(): + yield p.string('#') + s = yield p.regex('[^\\n]*') + yield p.string('\n') + return '#' + s + '\n' + +pws = p.regex('\s+') | pcomment1 | pcomment2 | pcomment3 + +@p.generate +def pstring(): + q = yield p.regex('["\']') + s = yield (p.string('\\%s' % q) | p.regex('[^%s]' % q)).many() + yield p.string(q) + return q + ''.join(s) + q + +@p.generate +def pnested(): + l = yield p.string('(') + n = yield pexpr.until(p.string(')')) + r = yield p.string(')') + return l + ''.join(n) + r + +pexpr = ( + # shortcut for a bit better performance + p.regex('[^%s/#\'"();{}=><,&|-]+' % ASSERT_CHARS) | + pws | + passert | + pstring | + pnested | + p.string('->') | + p.regex('.', re.DOTALL)) + +@p.generate +def pstmt(): + ws = yield pws.many() + lh = yield pexpr.until(p.string('=>') | p.regex('[;{}]')) + op = yield p.string('=>').optional() + if op == '=>': + rh = yield pstmt + return ''.join(ws) + mkassert(''.join(lh), rh, 'int', 'eq') + else: + return ''.join(ws) + ''.join(lh) + +@p.generate +def pstmts(): + a = yield pstmt + b = yield (p.regex('[;{}]') + pstmt).many() + return [a] + b + +def main(args): + inf = open(args.input, 'r') if args.input else sys.stdin + outf = open(args.output, 'w') if args.output else sys.stdout + + # parse C code + input = inf.read() + stmts = pstmts.parse(input) + + # write extra verbose asserts + outf.write("#include \n") + outf.write("#include \n") + outf.write("#include \n") + outf.write(mkdecl('int', 'eq', '==')) + outf.write(mkdecl('int', 'ne', '!=')) + outf.write(mkdecl('int', 'lt', '<')) + outf.write(mkdecl('int', 'gt', '>')) + outf.write(mkdecl('int', 'le', '<=')) + outf.write(mkdecl('int', 'ge', '>=')) + outf.write(mkdecl('str', 'eq', '==')) + outf.write(mkdecl('str', 'ne', '!=')) + outf.write(mkdecl('str', 'lt', '<')) + outf.write(mkdecl('str', 'gt', '>')) + outf.write(mkdecl('str', 'le', '<=')) + outf.write(mkdecl('str', 'ge', '>=')) + outf.write(mkdecl('bool', 'eq', '==')) + if args.input: + outf.write("#line %d \"%s\"\n" % (1, args.input)) + + # write parsed statements + for stmt in stmts: + outf.write(stmt) + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser( + description="Cpp step that increases assert verbosity") + parser.add_argument('input', nargs='?', + help="Input C file after cpp.") + parser.add_argument('-o', '--output', + help="Output C file.") + main(parser.parse_args()) diff --git a/scripts/test_.py b/scripts/test_.py new file mode 100755 index 00000000..f3f3b011 --- /dev/null +++ b/scripts/test_.py @@ -0,0 +1,494 @@ +#!/usr/bin/env python3 + +# TODO +# -v --verbose +# --color +# --gdb +# --reentrant + +import toml +import glob +import re +import os +import io +import itertools as it +import collections.abc as abc +import subprocess as sp +import base64 +import sys +import copy + +TEST_DIR = 'tests_' + +RULES = """ +define FLATTEN +%$(subst /,.,$(target:.c=.t.c)): $(target) + cat <(echo '#line 1 "$$<"') $$< > $$@ +endef +$(foreach target,$(SRC),$(eval $(FLATTEN))) + +-include tests_/*.d + +%.c: %.t.c + ./scripts/explode_asserts.py $< -o $@ + +%.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.test.$f) + $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ +""" +GLOBALS = """ +//////////////// AUTOGENERATED TEST //////////////// +#include "lfs.h" +#include "emubd/lfs_emubd.h" +#include +""" +DEFINES = { + "LFS_READ_SIZE": 16, + "LFS_PROG_SIZE": "LFS_READ_SIZE", + "LFS_BLOCK_SIZE": 512, + "LFS_BLOCK_COUNT": 1024, + "LFS_BLOCK_CYCLES": 1024, + "LFS_CACHE_SIZE": "(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)", + "LFS_LOOKAHEAD_SIZE": 16, +} +PROLOGUE = """ + // prologue + __attribute__((unused)) lfs_t lfs; + __attribute__((unused)) lfs_emubd_t bd; + __attribute__((unused)) lfs_file_t file; + __attribute__((unused)) lfs_dir_t dir; + __attribute__((unused)) struct lfs_info info; + __attribute__((unused)) uint8_t buffer[1024]; + __attribute__((unused)) char path[1024]; + + __attribute__((unused)) const struct lfs_config cfg = { + .context = &bd, + .read = &lfs_emubd_read, + .prog = &lfs_emubd_prog, + .erase = &lfs_emubd_erase, + .sync = &lfs_emubd_sync, + + .read_size = LFS_READ_SIZE, + .prog_size = LFS_PROG_SIZE, + .block_size = LFS_BLOCK_SIZE, + .block_count = LFS_BLOCK_COUNT, + .block_cycles = LFS_BLOCK_CYCLES, + .cache_size = LFS_CACHE_SIZE, + .lookahead_size = LFS_LOOKAHEAD_SIZE, + }; + + lfs_emubd_create(&cfg, "blocks"); +""" +EPILOGUE = """ + // epilogue + lfs_emubd_destroy(&cfg); +""" +PASS = '\033[32m✓\033[0m' +FAIL = '\033[31m✗\033[0m' + +class TestFailure(Exception): + def __init__(self, case, stdout=None, assert_=None): + self.case = case + self.stdout = stdout + self.assert_ = assert_ + +class TestCase: + def __init__(self, suite, config, caseno=None, lineno=None, **_): + self.suite = suite + self.caseno = caseno + self.lineno = lineno + + self.code = config['code'] + self.defines = config.get('define', {}) + self.leaky = config.get('leaky', False) + + def __str__(self): + if hasattr(self, 'permno'): + return '%s[%d,%d]' % (self.suite.name, self.caseno, self.permno) + else: + return '%s[%d]' % (self.suite.name, self.caseno) + + def permute(self, defines, permno=None, **_): + ncase = copy.copy(self) + ncase.case = self + ncase.perms = [ncase] + ncase.permno = permno + ncase.defines = defines + return ncase + + def build(self, f, **_): + # prologue + f.write('void test_case%d(' % self.caseno) + defines = self.perms[0].defines + first = True + for k, v in sorted(defines.items()): + if not all(perm.defines[k] == v for perm in self.perms): + if not first: + f.write(',') + else: + first = False + f.write('\n') + f.write(8*' '+'int %s' % k) + f.write(') {\n') + + defines = self.perms[0].defines + for k, v in sorted(defines.items()): + if all(perm.defines[k] == v for perm in self.perms): + f.write(4*' '+'#define %s %s\n' % (k, v)) + + f.write(PROLOGUE) + f.write('\n') + f.write(4*' '+'// test case %d\n' % self.caseno) + f.write(4*' '+'#line %d "%s"\n' % (self.lineno, self.suite.path)) + + # test case goes here + f.write(self.code) + + # epilogue + f.write(EPILOGUE) + f.write('\n') + + defines = self.perms[0].defines + for k, v in sorted(defines.items()): + if all(perm.defines[k] == v for perm in self.perms): + f.write(4*' '+'#undef %s\n' % k) + + f.write('}\n') + + def test(self, **args): + cmd = ['./%s.test' % self.suite.path, + repr(self.caseno), repr(self.permno)] + + # run in valgrind? + if args.get('valgrind', False) and not self.leaky: + cmd = ['valgrind', + '--leak-check=full', + '--error-exitcode=4', + '-q'] + cmd + + # run test case! + stdout = [] + if args.get('verbose', False): + print(' '.join(cmd)) + proc = sp.Popen(cmd, + universal_newlines=True, + bufsize=1, + stdout=sp.PIPE, + stderr=sp.STDOUT) + for line in iter(proc.stdout.readline, ''): + stdout.append(line) + if args.get('verbose', False): + sys.stdout.write(line) + proc.wait() + + if proc.returncode != 0: + # failed, try to parse assert? + assert_ = None + for line in stdout: + try: + m = re.match('^([^:\\n]+):([0-9]+):assert: (.*)$', line) + # found an assert, print info from file + with open(m.group(1)) as f: + lineno = int(m.group(2)) + line = next(it.islice(f, lineno-1, None)).strip('\n') + assert_ = { + 'path': m.group(1), + 'lineno': lineno, + 'line': line, + 'message': m.group(3), + } + except: + pass + + self.result = TestFailure(self, stdout, assert_) + raise self.result + + else: + self.result = PASS + return self.result + +class TestSuite: + def __init__(self, path, TestCase=TestCase, **args): + self.name = os.path.basename(path) + if self.name.endswith('.toml'): + self.name = self.name[:-len('.toml')] + self.path = path + self.TestCase = TestCase + + with open(path) as f: + # load tests + config = toml.load(f) + + # find line numbers + f.seek(0) + linenos = [] + for i, line in enumerate(f): + if re.match(r'^\s*code\s*=\s*(\'\'\'|""")', line): + linenos.append(i + 2) + + # grab global config + self.defines = config.get('define', {}) + + # create initial test cases + self.cases = [] + for i, (case, lineno) in enumerate(zip(config['case'], linenos)): + self.cases.append(self.TestCase( + self, case, caseno=i, lineno=lineno, **args)) + + def __str__(self): + return self.name + + def __lt__(self, other): + return self.name < other.name + + def permute(self, defines={}, **args): + for case in self.cases: + # lets find all parameterized definitions, in one of + # - args.D (defines) + # - suite.defines + # - case.defines + # - DEFINES + initial = {} + for define in it.chain( + defines.items(), + self.defines.items(), + case.defines.items(), + DEFINES.items()): + if define[0] not in initial: + try: + initial[define[0]] = eval(define[1]) + except: + initial[define[0]] = define[1] + + # expand permutations + expanded = [] + pending = [initial] + while pending: + perm = pending.pop() + for k, v in sorted(perm.items()): + if not isinstance(v, str) and isinstance(v, abc.Iterable): + for nv in reversed(v): + nperm = perm.copy() + nperm[k] = nv + pending.append(nperm) + break + else: + expanded.append(perm) + + case.perms = [] + for i, defines in enumerate(expanded): + case.perms.append(case.permute(defines, permno=i, **args)) + + self.perms = [perm for case in self.cases for perm in case.perms] + return self.perms + + def build(self, **args): + # build test.c + f = io.StringIO() + f.write(GLOBALS) + + for case in self.cases: + f.write('\n') + case.build(f, **args) + + f.write('\n') + f.write('int main(int argc, char **argv) {\n') + f.write(4*' '+'int case_ = (argc == 3) ? atoi(argv[1]) : 0;\n') + f.write(4*' '+'int perm = (argc == 3) ? atoi(argv[2]) : 0;\n') + for perm in self.perms: + f.write(4*' '+'if (argc != 3 || ' + '(case_ == %d && perm == %d)) { ' % ( + perm.caseno, perm.permno)) + f.write('test_case%d(' % perm.caseno) + first = True + for k, v in sorted(perm.defines.items()): + if not all(perm.defines[k] == v for perm in perm.case.perms): + if not first: + f.write(', ') + else: + first = False + f.write(str(v)) + f.write('); }\n') + f.write('}\n') + + # add test-related rules + rules = RULES + rules = rules.replace(' ', '\t') + + with open(self.path + '.test.mk', 'w') as mk: + mk.write(rules) + mk.write('\n') + + mk.write('%s: %s\n' % (self.path+'.test.t.c', self.path)) + mk.write('\tbase64 -d <<< ') + mk.write(base64.b64encode( + f.getvalue().encode('utf8')).decode('utf8')) + mk.write(' > $@\n') + + self.makefile = self.path + '.test.mk' + self.target = self.path + '.test' + return self.makefile, self.target + + def test(self, caseno=None, permno=None, **args): + # run test suite! + if not args.get('verbose', True): + sys.stdout.write(self.name + ' ') + sys.stdout.flush() + for perm in self.perms: + if caseno is not None and perm.caseno != caseno: + continue + if permno is not None and perm.permno != permno: + continue + + try: + perm.test(**args) + except TestFailure as failure: + if not args.get('verbose', True): + sys.stdout.write(FAIL) + sys.stdout.flush() + if not args.get('keep_going', False): + if not args.get('verbose', True): + sys.stdout.write('\n') + raise + else: + if not args.get('verbose', True): + sys.stdout.write(PASS) + sys.stdout.flush() + + if not args.get('verbose', True): + sys.stdout.write('\n') + +def main(**args): + testpath = args['testpath'] + + # optional brackets for specific test + m = re.search(r'\[(\d+)(?:,(\d+))?\]$', testpath) + if m: + caseno = int(m.group(1)) + permno = int(m.group(2)) if m.group(2) is not None else None + testpath = testpath[:m.start()] + else: + caseno = None + permno = None + + # figure out the suite's toml file + if os.path.isdir(testpath): + testpath = testpath + '/test_*.toml' + elif os.path.isfile(testpath): + testpath = testpath + elif testpath.endswith('.toml'): + testpath = TEST_DIR + '/' + testpath + else: + testpath = TEST_DIR + '/' + testpath + '.toml' + + # find tests + suites = [] + for path in glob.glob(testpath): + suites.append(TestSuite(path, **args)) + + # sort for reproducability + suites = sorted(suites) + + # generate permutations + defines = {} + for define in args['D']: + k, v, *_ = define.split('=', 2) + [''] + defines[k] = v + + for suite in suites: + suite.permute(defines, **args) + + # build tests in parallel + print('====== building ======') + makefiles = [] + targets = [] + for suite in suites: + makefile, target = suite.build(**args) + makefiles.append(makefile) + targets.append(target) + + cmd = (['make', '-f', 'Makefile'] + + list(it.chain.from_iterable(['-f', m] for m in makefiles)) + + ['CFLAGS+=-fdiagnostics-color=always'] + + [target for target in targets]) + stdout = [] + if args.get('verbose', False): + print(' '.join(cmd)) + proc = sp.Popen(cmd, + universal_newlines=True, + bufsize=1, + stdout=sp.PIPE, + stderr=sp.STDOUT) + for line in iter(proc.stdout.readline, ''): + stdout.append(line) + if args.get('verbose', False): + sys.stdout.write(line) + proc.wait() + + if proc.returncode != 0: + if not args.get('verbose', False): + for line in stdout: + sys.stdout.write(line) + sys.exit(-3) + + print('built %d test suites, %d test cases, %d permutations' % ( + len(suites), + sum(len(suite.cases) for suite in suites), + sum(len(suite.perms) for suite in suites))) + + print('====== testing ======') + try: + for suite in suites: + suite.test(caseno, permno, **args) + except TestFailure: + pass + + print('====== results ======') + passed = 0 + failed = 0 + for suite in suites: + for perm in suite.perms: + if not hasattr(perm, 'result'): + continue + + if perm.result == PASS: + passed += 1 + else: + sys.stdout.write("--- %s ---\n" % perm) + if perm.result.assert_: + for line in perm.result.stdout[:-1]: + sys.stdout.write(line) + sys.stdout.write( + "\033[97m{path}:{lineno}:\033[91massert:\033[0m " + "{message}\n{line}\n".format( + **perm.result.assert_)) + else: + for line in perm.result.stdout: + sys.stdout.write(line) + sys.stdout.write('\n') + failed += 1 + + print('tests passed: %d' % passed) + print('tests failed: %d' % failed) + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser( + description="Run parameterized tests in various configurations.") + parser.add_argument('testpath', nargs='?', default=TEST_DIR, + help="Description of test(s) to run. By default, this is all tests \ + found in the \"{0}\" directory. Here, you can specify a different \ + directory of tests, a specific file, a suite by name, and even a \ + specific test case by adding brackets. For example \ + \"test_dirs[0]\" or \"{0}/test_dirs.toml[0]\".".format(TEST_DIR)) + parser.add_argument('-D', action='append', default=[], + help="Overriding parameter definitions.") + parser.add_argument('-v', '--verbose', action='store_true', + help="Output everything that is happening.") + parser.add_argument('-k', '--keep-going', action='store_true', + help="Run all tests instead of stopping on first error. Useful for CI.") +# parser.add_argument('--gdb', action='store_true', +# help="Run tests under gdb. Useful for debugging failures.") + parser.add_argument('--valgrind', action='store_true', + help="Run non-leaky tests under valgrind to check for memory leaks. \ + Tests marked as \"leaky = true\" run normally.") + main(**vars(parser.parse_args())) diff --git a/tests_/test_dirs.toml b/tests_/test_dirs.toml new file mode 100644 index 00000000..2e972815 --- /dev/null +++ b/tests_/test_dirs.toml @@ -0,0 +1,209 @@ +[[case]] # format +code = """ + lfs_format(&lfs, &cfg) => 0; +""" + +[[case]] # mount/unmount +code = """ + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_unmount(&lfs) => 0; +""" + +[[case]] # root +code = """ + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +""" + +[[case]] # directory creation +code = """ + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "dir%03d", i); + lfs_mkdir(&lfs, path) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "dir%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); +""" +define.N = 'range(0, 100, 3)' + +[[case]] # directory removal +code = """ + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "removeme%03d", i); + lfs_mkdir(&lfs, path) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "removeme%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "removeme%03d", i); + lfs_remove(&lfs, path) => 0; + } + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +""" +define.N = 'range(3, 100, 11)' + +[[case]] # file creation +code = """ + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "file%03d", i); + lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "file%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); +""" +define.N = 'range(3, 100, 11)' + +[[case]] # file removal +code = """ + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "removeme%03d", i); + lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "removeme%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "removeme%03d", i); + lfs_remove(&lfs, path) => 0; + } + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +""" +define.N = 'range(0, 100, 3)' + +[[case]] # error cases +code = """ + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "potato") => 0; + lfs_file_open(&lfs, &file, "burito", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "potato") => LFS_ERR_EXIST; + lfs_dir_open(&lfs, &dir, "tomato") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "burito") => LFS_ERR_NOTDIR; + lfs_file_open(&lfs, &file, "tomato", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "potato", LFS_O_RDONLY) => LFS_ERR_ISDIR; + lfs_unmount(&lfs) => 0; +""" From ed8341ec4cd171cf42c743121f311bb51014e573 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Mon, 30 Dec 2019 13:01:08 -0600 Subject: [PATCH 02/41] Reworked permutation generation in test framework and cleanup - Reworked how permutations work - Now with global defines as well (apply to all code) - Also supports lists of different permutation sets - Added better cleanup in tests and "make clean" --- Makefile | 1 + scripts/test_.py | 109 ++++++++++++++++++++++++++++++----------------- 2 files changed, 70 insertions(+), 40 deletions(-) diff --git a/Makefile b/Makefile index e17b427f..6a6ec647 100644 --- a/Makefile +++ b/Makefile @@ -90,3 +90,4 @@ clean: rm -f $(OBJ) rm -f $(DEP) rm -f $(ASM) + rm -f tests_/test_*.toml.* diff --git a/scripts/test_.py b/scripts/test_.py index f3f3b011..736bf055 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -1,10 +1,8 @@ #!/usr/bin/env python3 -# TODO -# -v --verbose -# --color -# --gdb -# --reentrant +# This script manages littlefs tests, which are configured with +# .toml files stored in the tests directory. +# import toml import glob @@ -17,6 +15,7 @@ import base64 import sys import copy +import shutil TEST_DIR = 'tests_' @@ -118,22 +117,19 @@ def permute(self, defines, permno=None, **_): def build(self, f, **_): # prologue f.write('void test_case%d(' % self.caseno) - defines = self.perms[0].defines first = True - for k, v in sorted(defines.items()): - if not all(perm.defines[k] == v for perm in self.perms): + for k, v in sorted(self.perms[0].defines.items()): + if k not in self.defines: if not first: f.write(',') else: first = False f.write('\n') - f.write(8*' '+'int %s' % k) + f.write(8*' '+'__attribute__((unused)) intmax_t %s' % k) f.write(') {\n') - defines = self.perms[0].defines - for k, v in sorted(defines.items()): - if all(perm.defines[k] == v for perm in self.perms): - f.write(4*' '+'#define %s %s\n' % (k, v)) + for k, v in sorted(self.defines.items()): + f.write(4*' '+'#define %s %s\n' % (k, v)) f.write(PROLOGUE) f.write('\n') @@ -147,14 +143,16 @@ def build(self, f, **_): f.write(EPILOGUE) f.write('\n') - defines = self.perms[0].defines - for k, v in sorted(defines.items()): - if all(perm.defines[k] == v for perm in self.perms): - f.write(4*' '+'#undef %s\n' % k) + for k, v in sorted(self.defines.items()): + f.write(4*' '+'#undef %s\n' % k) f.write('}\n') def test(self, **args): + # clear disk first + shutil.rmtree('blocks') + + # build command cmd = ['./%s.test' % self.suite.path, repr(self.caseno), repr(self.permno)] @@ -242,26 +240,31 @@ def __lt__(self, other): def permute(self, defines={}, **args): for case in self.cases: - # lets find all parameterized definitions, in one of - # - args.D (defines) - # - suite.defines - # - case.defines - # - DEFINES - initial = {} - for define in it.chain( - defines.items(), - self.defines.items(), - case.defines.items(), - DEFINES.items()): - if define[0] not in initial: - try: - initial[define[0]] = eval(define[1]) - except: - initial[define[0]] = define[1] + # lets find all parameterized definitions, in one of [args.D, + # suite.defines, case.defines, DEFINES]. Note that each of these + # can be either a dict of defines, or a list of dicts, expressing + # an initial set of permutations. + pending = [{}] + for inits in [defines, self.defines, case.defines, DEFINES]: + if not isinstance(inits, list): + inits = [inits] + + npending = [] + for init, pinit in it.product(inits, pending): + ninit = pinit.copy() + for k, v in init.items(): + if k not in ninit: + try: + ninit[k] = eval(v) + except: + ninit[k] = v + npending.append(ninit) + + pending = npending # expand permutations + pending = list(reversed(pending)) expanded = [] - pending = [initial] while pending: perm = pending.pop() for k, v in sorted(perm.items()): @@ -274,11 +277,27 @@ def permute(self, defines={}, **args): else: expanded.append(perm) + # generate permutations case.perms = [] - for i, defines in enumerate(expanded): - case.perms.append(case.permute(defines, permno=i, **args)) + for i, perm in enumerate(expanded): + case.perms.append(case.permute(perm, permno=i, **args)) + + # also track non-unique defines + case.defines = {} + for k, v in case.perms[0].defines.items(): + if all(perm.defines[k] == v for perm in case.perms): + case.defines[k] = v + + # track all perms and non-unique defines + self.perms = [] + for case in self.cases: + self.perms.extend(case.perms) + + self.defines = {} + for k, v in self.perms[0].defines.items(): + if all(perm.defines[k] == v for perm in self.perms): + self.defines[k] = v - self.perms = [perm for case in self.cases for perm in case.perms] return self.perms def build(self, **args): @@ -301,7 +320,7 @@ def build(self, **args): f.write('test_case%d(' % perm.caseno) first = True for k, v in sorted(perm.defines.items()): - if not all(perm.defines[k] == v for perm in perm.case.perms): + if k not in perm.case.defines: if not first: f.write(', ') else: @@ -311,13 +330,18 @@ def build(self, **args): f.write('}\n') # add test-related rules - rules = RULES - rules = rules.replace(' ', '\t') + rules = RULES.replace(4*' ', '\t') with open(self.path + '.test.mk', 'w') as mk: mk.write(rules) mk.write('\n') + # add truely global defines globally + for k, v in sorted(self.defines.items()): + mk.write('%s: override CFLAGS += -D%s=%r\n' % ( + self.path+'.test', k, v)) + + # write test.c in base64 so make can decide when to rebuild mk.write('%s: %s\n' % (self.path+'.test.t.c', self.path)) mk.write('\tbase64 -d <<< ') mk.write(base64.b64encode( @@ -484,8 +508,13 @@ def main(**args): help="Overriding parameter definitions.") parser.add_argument('-v', '--verbose', action='store_true', help="Output everything that is happening.") + parser.add_argument('-t', '--trace', action='store_true', + help="Normally trace output is captured for internal usage, this \ + enables forwarding trace output which is usually too verbose to \ + be useful.") parser.add_argument('-k', '--keep-going', action='store_true', help="Run all tests instead of stopping on first error. Useful for CI.") +# TODO # parser.add_argument('--gdb', action='store_true', # help="Run tests under gdb. Useful for debugging failures.") parser.add_argument('--valgrind', action='store_true', From 53d2b02f2a113326611b24c069d6098e9cdf53c4 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 31 Dec 2019 11:51:52 -0600 Subject: [PATCH 03/41] Added reentrant and gdb testing mechanisms to test framework Aside from reworking the internals of test_.py to work well with inherited TestCase classes, this also provides the two main features that were the main reason for revamping the test framework 1. ./scripts/test_.py --reentrant Runs reentrant tests (tests with reentrant=true in the .toml configuration) under gdb such that the program is killed on every call to lfs_emubd_prog or lfs_emubd_erase. Currently this just increments a number of prog/erases to skip, which means it doesn't necessarily check every possible branch of the test, but this should still provide a good coverage of power-loss tests. 2. ./scripts/test_.py --gdb Run the tests and if a failure is hit, drop into GDB. In theory this will be very useful for reproducing and debugging test failures. Note this can be combined with --reentrant to drop into GDB on the exact cycle of power-loss where the tests fail. --- scripts/explode_asserts.py | 10 +- scripts/test_.py | 214 +++++++++++++++++++++++++------------ tests_/test_dirs.toml | 13 ++- 3 files changed, 165 insertions(+), 72 deletions(-) diff --git a/scripts/explode_asserts.py b/scripts/explode_asserts.py index 4e56710e..7c24c633 100755 --- a/scripts/explode_asserts.py +++ b/scripts/explode_asserts.py @@ -16,7 +16,8 @@ printf("%s:%d:assert: " "assert failed with %"PRIiMAX", expected {comp} %"PRIiMAX"\\n", {file}, {line}, (intmax_t)_lh, (intmax_t)_rh); - exit(-2); + fflush(NULL); + raise(SIGABRT); }} """, 'str': """ @@ -26,7 +27,8 @@ printf("%s:%d:assert: " "assert failed with \\\"%s\\\", expected {comp} \\\"%s\\\"\\n", {file}, {line}, _lh, _rh); - exit(-2); + fflush(NULL); + raise(SIGABRT); }} """, 'bool': """ @@ -36,7 +38,8 @@ printf("%s:%d:assert: " "assert failed with %s, expected {comp} %s\\n", {file}, {line}, _lh ? "true" : "false", _rh ? "true" : "false"); - exit(-2); + fflush(NULL); + raise(SIGABRT); }} """, } @@ -180,6 +183,7 @@ def main(args): outf.write("#include \n") outf.write("#include \n") outf.write("#include \n") + outf.write("#include \n") outf.write(mkdecl('int', 'eq', '==')) outf.write(mkdecl('int', 'ne', '!=')) outf.write(mkdecl('int', 'lt', '<')) diff --git a/scripts/test_.py b/scripts/test_.py index 736bf055..5a481e40 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -16,9 +16,9 @@ import sys import copy import shutil +import shlex -TEST_DIR = 'tests_' - +TESTDIR = 'tests_' RULES = """ define FLATTEN %$(subst /,.,$(target:.c=.t.c)): $(target) @@ -28,9 +28,12 @@ -include tests_/*.d +.SECONDARY: %.c: %.t.c ./scripts/explode_asserts.py $< -o $@ +%.test: override CFLAGS += -fdiagnostics-color=always +%.test: override CFLAGS += -ggdb %.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.test.$f) $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ """ @@ -60,18 +63,18 @@ __attribute__((unused)) char path[1024]; __attribute__((unused)) const struct lfs_config cfg = { - .context = &bd, - .read = &lfs_emubd_read, - .prog = &lfs_emubd_prog, - .erase = &lfs_emubd_erase, - .sync = &lfs_emubd_sync, - - .read_size = LFS_READ_SIZE, - .prog_size = LFS_PROG_SIZE, - .block_size = LFS_BLOCK_SIZE, - .block_count = LFS_BLOCK_COUNT, - .block_cycles = LFS_BLOCK_CYCLES, - .cache_size = LFS_CACHE_SIZE, + .context = &bd, + .read = &lfs_emubd_read, + .prog = &lfs_emubd_prog, + .erase = &lfs_emubd_erase, + .sync = &lfs_emubd_sync, + + .read_size = LFS_READ_SIZE, + .prog_size = LFS_PROG_SIZE, + .block_size = LFS_BLOCK_SIZE, + .block_count = LFS_BLOCK_COUNT, + .block_cycles = LFS_BLOCK_CYCLES, + .cache_size = LFS_CACHE_SIZE, .lookahead_size = LFS_LOOKAHEAD_SIZE, }; @@ -85,13 +88,14 @@ FAIL = '\033[31m✗\033[0m' class TestFailure(Exception): - def __init__(self, case, stdout=None, assert_=None): + def __init__(self, case, returncode=None, stdout=None, assert_=None): self.case = case + self.returncode = returncode self.stdout = stdout self.assert_ = assert_ class TestCase: - def __init__(self, suite, config, caseno=None, lineno=None, **_): + def __init__(self, config, suite=None, caseno=None, lineno=None, **_): self.suite = suite self.caseno = caseno self.lineno = lineno @@ -148,25 +152,29 @@ def build(self, f, **_): f.write('}\n') - def test(self, **args): + def test(self, exec=[], persist=False, gdb=False, failure=None, **args): # clear disk first - shutil.rmtree('blocks') + if not persist: + shutil.rmtree('blocks', True) # build command - cmd = ['./%s.test' % self.suite.path, + cmd = exec + ['./%s.test' % self.suite.path, repr(self.caseno), repr(self.permno)] - # run in valgrind? - if args.get('valgrind', False) and not self.leaky: - cmd = ['valgrind', - '--leak-check=full', - '--error-exitcode=4', - '-q'] + cmd + # failed? drop into debugger? + if gdb and failure: + cmd = (['gdb', '-ex', 'r' + ] + (['-ex', 'up'] if failure.assert_ else []) + [ + '--args'] + cmd) + if args.get('verbose', False): + print(' '.join(shlex.quote(c) for c in cmd)) + sys.exit(sp.call(cmd)) # run test case! stdout = [] + assert_ = None if args.get('verbose', False): - print(' '.join(cmd)) + print(' '.join(shlex.quote(c) for c in cmd)) proc = sp.Popen(cmd, universal_newlines=True, bufsize=1, @@ -176,33 +184,84 @@ def test(self, **args): stdout.append(line) if args.get('verbose', False): sys.stdout.write(line) - proc.wait() - - if proc.returncode != 0: - # failed, try to parse assert? - assert_ = None - for line in stdout: + # intercept asserts + m = re.match('^([^:]+):([0-9]+):(assert): (.*)$', line) + if m and assert_ is None: try: - m = re.match('^([^:\\n]+):([0-9]+):assert: (.*)$', line) - # found an assert, print info from file with open(m.group(1)) as f: lineno = int(m.group(2)) line = next(it.islice(f, lineno-1, None)).strip('\n') - assert_ = { - 'path': m.group(1), - 'lineno': lineno, - 'line': line, - 'message': m.group(3), - } + assert_ = { + 'path': m.group(1), + 'line': line, + 'lineno': lineno, + 'message': m.group(4)} except: pass + proc.wait() - self.result = TestFailure(self, stdout, assert_) - raise self.result - + # did we pass? + if proc.returncode != 0: + raise TestFailure(self, proc.returncode, stdout, assert_) else: - self.result = PASS - return self.result + return PASS + +class ValgrindTestCase(TestCase): + def __init__(self, config, **args): + self.leaky = config.get('leaky', False) + super().__init__(config, **args) + + def test(self, exec=[], **args): + if self.leaky: + return + + exec = exec + [ + 'valgrind', + '--leak-check=full', + '--error-exitcode=4', + '-q'] + return super().test(exec=exec, **args) + +class ReentrantTestCase(TestCase): + def __init__(self, config, **args): + self.reentrant = config.get('reentrant', False) + super().__init__(config, **args) + + def test(self, exec=[], persist=False, gdb=False, failure=None, **args): + if not self.reentrant: + return + + for cycles in it.count(1): + npersist = persist or cycles > 1 + + # exact cycle we should drop into debugger? + if gdb and failure and failure.cycleno == cycles: + return super().test(exec=exec, persist=npersist, + gdb=gdb, failure=failure, **args) + + # run tests, but kill the program after lfs_emubd_prog/erase has + # been hit n cycles. We exit with a special return code if the + # program has not finished, since this isn't a test failure. + nexec = exec + [ + 'gdb', '-batch-silent', + '-ex', 'handle all nostop', + '-ex', 'b lfs_emubd_prog', + '-ex', 'b lfs_emubd_erase', + '-ex', 'r', + ] + cycles*['-ex', 'c'] + [ + '-ex', 'q ' + '!$_isvoid($_exitsignal) ? $_exitsignal : ' + '!$_isvoid($_exitcode) ? $_exitcode : ' + '33', + '--args'] + try: + return super().test(exec=nexec, persist=npersist, **args) + except TestFailure as nfailure: + if nfailure.returncode == 33: + continue + else: + nfailure.cycleno = cycles + raise class TestSuite: def __init__(self, path, TestCase=TestCase, **args): @@ -229,8 +288,8 @@ def __init__(self, path, TestCase=TestCase, **args): # create initial test cases self.cases = [] for i, (case, lineno) in enumerate(zip(config['case'], linenos)): - self.cases.append(self.TestCase( - self, case, caseno=i, lineno=lineno, **args)) + self.cases.append(self.TestCase(case, + suite=self, caseno=i, lineno=lineno, **args)) def __str__(self): return self.name @@ -343,7 +402,7 @@ def build(self, **args): # write test.c in base64 so make can decide when to rebuild mk.write('%s: %s\n' % (self.path+'.test.t.c', self.path)) - mk.write('\tbase64 -d <<< ') + mk.write('\t@base64 -d <<< ') mk.write(base64.b64encode( f.getvalue().encode('utf8')).decode('utf8')) mk.write(' > $@\n') @@ -364,8 +423,9 @@ def test(self, caseno=None, permno=None, **args): continue try: - perm.test(**args) + result = perm.test(**args) except TestFailure as failure: + perm.result = failure if not args.get('verbose', True): sys.stdout.write(FAIL) sys.stdout.flush() @@ -374,9 +434,11 @@ def test(self, caseno=None, permno=None, **args): sys.stdout.write('\n') raise else: - if not args.get('verbose', True): - sys.stdout.write(PASS) - sys.stdout.flush() + if result == PASS: + perm.result = PASS + if not args.get('verbose', True): + sys.stdout.write(PASS) + sys.stdout.flush() if not args.get('verbose', True): sys.stdout.write('\n') @@ -400,14 +462,19 @@ def main(**args): elif os.path.isfile(testpath): testpath = testpath elif testpath.endswith('.toml'): - testpath = TEST_DIR + '/' + testpath + testpath = TESTDIR + '/' + testpath else: - testpath = TEST_DIR + '/' + testpath + '.toml' + testpath = TESTDIR + '/' + testpath + '.toml' # find tests suites = [] for path in glob.glob(testpath): - suites.append(TestSuite(path, **args)) + if args.get('valgrind', False): + suites.append(TestSuite(path, TestCase=ValgrindTestCase, **args)) + elif args.get('reentrant', False): + suites.append(TestSuite(path, TestCase=ReentrantTestCase, **args)) + else: + suites.append(TestSuite(path, **args)) # sort for reproducability suites = sorted(suites) @@ -432,11 +499,10 @@ def main(**args): cmd = (['make', '-f', 'Makefile'] + list(it.chain.from_iterable(['-f', m] for m in makefiles)) + - ['CFLAGS+=-fdiagnostics-color=always'] + [target for target in targets]) stdout = [] if args.get('verbose', False): - print(' '.join(cmd)) + print(' '.join(shlex.quote(c) for c in cmd)) proc = sp.Popen(cmd, universal_newlines=True, bufsize=1, @@ -466,6 +532,18 @@ def main(**args): except TestFailure: pass + if args.get('gdb', False): + failure = None + for suite in suites: + for perm in suite.perms: + if getattr(perm, 'result', PASS) != PASS: + failure = perm.result + if failure is not None: + print('======= gdb ======') + # drop into gdb + failure.case.test(failure=failure, **args) + sys.exit(0) + print('====== results ======') passed = 0 failed = 0 @@ -498,26 +576,26 @@ def main(**args): import argparse parser = argparse.ArgumentParser( description="Run parameterized tests in various configurations.") - parser.add_argument('testpath', nargs='?', default=TEST_DIR, + parser.add_argument('testpath', nargs='?', default=TESTDIR, help="Description of test(s) to run. By default, this is all tests \ found in the \"{0}\" directory. Here, you can specify a different \ directory of tests, a specific file, a suite by name, and even a \ specific test case by adding brackets. For example \ - \"test_dirs[0]\" or \"{0}/test_dirs.toml[0]\".".format(TEST_DIR)) + \"test_dirs[0]\" or \"{0}/test_dirs.toml[0]\".".format(TESTDIR)) parser.add_argument('-D', action='append', default=[], help="Overriding parameter definitions.") parser.add_argument('-v', '--verbose', action='store_true', help="Output everything that is happening.") - parser.add_argument('-t', '--trace', action='store_true', - help="Normally trace output is captured for internal usage, this \ - enables forwarding trace output which is usually too verbose to \ - be useful.") parser.add_argument('-k', '--keep-going', action='store_true', help="Run all tests instead of stopping on first error. Useful for CI.") -# TODO -# parser.add_argument('--gdb', action='store_true', -# help="Run tests under gdb. Useful for debugging failures.") + parser.add_argument('-p', '--persist', action='store_true', + help="Don't reset the tests disk before each test.") + parser.add_argument('-g', '--gdb', action='store_true', + help="Drop into gdb on failure.") parser.add_argument('--valgrind', action='store_true', - help="Run non-leaky tests under valgrind to check for memory leaks. \ - Tests marked as \"leaky = true\" run normally.") + help="Run non-leaky tests under valgrind to check for memory leaks.") + parser.add_argument('--reentrant', action='store_true', + help="Run reentrant tests with simulated power-loss.") + parser.add_argument('-e', '--exec', default=[], type=lambda e: e.split(' '), + help="Run tests with another executable prefixed on the command line.") main(**vars(parser.parse_args())) diff --git a/tests_/test_dirs.toml b/tests_/test_dirs.toml index 2e972815..41789d59 100644 --- a/tests_/test_dirs.toml +++ b/tests_/test_dirs.toml @@ -10,6 +10,17 @@ code = """ lfs_unmount(&lfs) => 0; """ +[[case]] # reentrant format +code = """ + int err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + lfs_unmount(&lfs) => 0; +""" +reentrant = true + [[case]] # root code = """ lfs_format(&lfs, &cfg) => 0; @@ -53,7 +64,7 @@ code = """ } lfs_dir_read(&lfs, &dir, &info) => 0; lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs); + lfs_unmount(&lfs) => 0; """ define.N = 'range(0, 100, 3)' From eeaf536ecabbc9a3813d068e367dbe7897e60bcf Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Thu, 2 Jan 2020 18:36:53 -0600 Subject: [PATCH 04/41] Replaced emubd with rambd and filebd The idea behind emubd (file per block), was neat, but doesn't add much value over a block device that just operates on a single linear file (other than adding a significant amount of overhead). Initially it helped with debugging, but when the metadata format became more complex in v2, most debugging ends up going through the debug.py script anyways. Aside from being simpler, moving to filebd means it is also possible to mount disk images directly. Also introduced rambd, which keeps the disk contents in RAM. This is very useful for testing where it increases the speed _significantly_. - test_dirs w/ filebd - 0m7.170s - test_dirs w/ rambd - 0m0.966s These follow the emubd model of using the lfs_config for geometry. I'm not convinced this is the best approach, but it gets the job done. I've also added lfs_ramdb_createcfg to add additional config similar to lfs_file_opencfg. This is useful for specifying erase_value, which tells the block device to simulate erases similar to flash devices. Note that the default (-1) meets the minimum block device requirements and is the most performant. --- Makefile | 2 +- emubd/lfs_emubd.c | 414 -------------------------------------------- emubd/lfs_emubd.h | 79 --------- filebd/lfs_filebd.c | 198 +++++++++++++++++++++ filebd/lfs_filebd.h | 64 +++++++ rambd/lfs_rambd.c | 137 +++++++++++++++ rambd/lfs_rambd.h | 67 +++++++ scripts/test_.py | 101 +++++++---- 8 files changed, 534 insertions(+), 528 deletions(-) delete mode 100644 emubd/lfs_emubd.c delete mode 100644 emubd/lfs_emubd.h create mode 100644 filebd/lfs_filebd.c create mode 100644 filebd/lfs_filebd.h create mode 100644 rambd/lfs_rambd.c create mode 100644 rambd/lfs_rambd.h diff --git a/Makefile b/Makefile index 6a6ec647..17d3ab38 100644 --- a/Makefile +++ b/Makefile @@ -7,7 +7,7 @@ CC ?= gcc AR ?= ar SIZE ?= size -SRC += $(wildcard *.c emubd/*.c) +SRC += $(wildcard *.c rambd/*.c filebd/*.c) OBJ := $(SRC:.c=.o) DEP := $(SRC:.c=.d) ASM := $(SRC:.c=.s) diff --git a/emubd/lfs_emubd.c b/emubd/lfs_emubd.c deleted file mode 100644 index 9a29ff1d..00000000 --- a/emubd/lfs_emubd.c +++ /dev/null @@ -1,414 +0,0 @@ -/* - * Block device emulated on standard files - * - * Copyright (c) 2017, Arm Limited. All rights reserved. - * SPDX-License-Identifier: BSD-3-Clause - */ -#include "emubd/lfs_emubd.h" - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - - -// Emulated block device utils -static inline void lfs_emubd_tole32(lfs_emubd_t *emu) { - emu->cfg.read_size = lfs_tole32(emu->cfg.read_size); - emu->cfg.prog_size = lfs_tole32(emu->cfg.prog_size); - emu->cfg.block_size = lfs_tole32(emu->cfg.block_size); - emu->cfg.block_count = lfs_tole32(emu->cfg.block_count); - - emu->stats.read_count = lfs_tole32(emu->stats.read_count); - emu->stats.prog_count = lfs_tole32(emu->stats.prog_count); - emu->stats.erase_count = lfs_tole32(emu->stats.erase_count); - - for (unsigned i = 0; i < sizeof(emu->history.blocks) / - sizeof(emu->history.blocks[0]); i++) { - emu->history.blocks[i] = lfs_tole32(emu->history.blocks[i]); - } -} - -static inline void lfs_emubd_fromle32(lfs_emubd_t *emu) { - emu->cfg.read_size = lfs_fromle32(emu->cfg.read_size); - emu->cfg.prog_size = lfs_fromle32(emu->cfg.prog_size); - emu->cfg.block_size = lfs_fromle32(emu->cfg.block_size); - emu->cfg.block_count = lfs_fromle32(emu->cfg.block_count); - - emu->stats.read_count = lfs_fromle32(emu->stats.read_count); - emu->stats.prog_count = lfs_fromle32(emu->stats.prog_count); - emu->stats.erase_count = lfs_fromle32(emu->stats.erase_count); - - for (unsigned i = 0; i < sizeof(emu->history.blocks) / - sizeof(emu->history.blocks[0]); i++) { - emu->history.blocks[i] = lfs_fromle32(emu->history.blocks[i]); - } -} - - -// Block device emulated on existing filesystem -int lfs_emubd_create(const struct lfs_config *cfg, const char *path) { - LFS_TRACE("lfs_emubd_create(%p {.context=%p, " - ".read=%p, .prog=%p, .erase=%p, .sync=%p, " - ".read_size=%"PRIu32", .prog_size=%"PRIu32", " - ".block_size=%"PRIu32", .block_count=%"PRIu32"}, \"%s\")", - (void*)cfg, cfg->context, - (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, - (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, - cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, - path); - lfs_emubd_t *emu = cfg->context; - emu->cfg.read_size = cfg->read_size; - emu->cfg.prog_size = cfg->prog_size; - emu->cfg.block_size = cfg->block_size; - emu->cfg.block_count = cfg->block_count; - - // Allocate buffer for creating children files - size_t pathlen = strlen(path); - emu->path = malloc(pathlen + 1 + LFS_NAME_MAX + 1); - if (!emu->path) { - int err = -ENOMEM; - LFS_TRACE("lfs_emubd_create -> %"PRId32, err); - return err; - } - - strcpy(emu->path, path); - emu->path[pathlen] = '/'; - emu->child = &emu->path[pathlen+1]; - memset(emu->child, '\0', LFS_NAME_MAX+1); - - // Create directory if it doesn't exist - int err = mkdir(path, 0777); - if (err && errno != EEXIST) { - err = -errno; - LFS_TRACE("lfs_emubd_create -> %"PRId32, err); - return err; - } - - // Load stats to continue incrementing - snprintf(emu->child, LFS_NAME_MAX, ".stats"); - FILE *f = fopen(emu->path, "r"); - if (!f) { - memset(&emu->stats, LFS_EMUBD_ERASE_VALUE, sizeof(emu->stats)); - } else { - size_t res = fread(&emu->stats, sizeof(emu->stats), 1, f); - lfs_emubd_fromle32(emu); - if (res < 1) { - err = -errno; - LFS_TRACE("lfs_emubd_create -> %"PRId32, err); - fclose(f); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_create -> %"PRId32, err); - return err; - } - } - - // Load history - snprintf(emu->child, LFS_NAME_MAX, ".history"); - f = fopen(emu->path, "r"); - if (!f) { - memset(&emu->history, 0, sizeof(emu->history)); - } else { - size_t res = fread(&emu->history, sizeof(emu->history), 1, f); - lfs_emubd_fromle32(emu); - if (res < 1) { - err = -errno; - LFS_TRACE("lfs_emubd_create -> %"PRId32, err); - fclose(f); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_create -> %"PRId32, err); - return err; - } - } - - LFS_TRACE("lfs_emubd_create -> %"PRId32, 0); - return 0; -} - -void lfs_emubd_destroy(const struct lfs_config *cfg) { - LFS_TRACE("lfs_emubd_destroy(%p)", (void*)cfg); - lfs_emubd_sync(cfg); - - lfs_emubd_t *emu = cfg->context; - free(emu->path); - LFS_TRACE("lfs_emubd_destroy -> %s", "void"); -} - -int lfs_emubd_read(const struct lfs_config *cfg, lfs_block_t block, - lfs_off_t off, void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_emubd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", - (void*)cfg, block, off, buffer, size); - lfs_emubd_t *emu = cfg->context; - uint8_t *data = buffer; - - // Check if read is valid - LFS_ASSERT(off % cfg->read_size == 0); - LFS_ASSERT(size % cfg->read_size == 0); - LFS_ASSERT(block < cfg->block_count); - - // Zero out buffer for debugging - memset(data, 0, size); - - // Read data - snprintf(emu->child, LFS_NAME_MAX, "%" PRIx32, block); - - FILE *f = fopen(emu->path, "rb"); - if (!f && errno != ENOENT) { - int err = -errno; - LFS_TRACE("lfs_emubd_read -> %d", err); - return err; - } - - if (f) { - int err = fseek(f, off, SEEK_SET); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_read -> %d", err); - fclose(f); - return err; - } - - size_t res = fread(data, 1, size, f); - if (res < size && !feof(f)) { - err = -errno; - LFS_TRACE("lfs_emubd_read -> %d", err); - fclose(f); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_read -> %d", err); - return err; - } - } - - emu->stats.read_count += size; - LFS_TRACE("lfs_emubd_read -> %d", 0); - return 0; -} - -int lfs_emubd_prog(const struct lfs_config *cfg, lfs_block_t block, - lfs_off_t off, const void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_emubd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", - (void*)cfg, block, off, buffer, size); - lfs_emubd_t *emu = cfg->context; - const uint8_t *data = buffer; - - // Check if write is valid - LFS_ASSERT(off % cfg->prog_size == 0); - LFS_ASSERT(size % cfg->prog_size == 0); - LFS_ASSERT(block < cfg->block_count); - - // Program data - snprintf(emu->child, LFS_NAME_MAX, "%" PRIx32, block); - - FILE *f = fopen(emu->path, "r+b"); - if (!f) { - int err = (errno == EACCES) ? 0 : -errno; - LFS_TRACE("lfs_emubd_prog -> %d", err); - return err; - } - - // Check that file was erased - LFS_ASSERT(f); - - int err = fseek(f, off, SEEK_SET); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_prog -> %d", err); - fclose(f); - return err; - } - - size_t res = fwrite(data, 1, size, f); - if (res < size) { - err = -errno; - LFS_TRACE("lfs_emubd_prog -> %d", err); - fclose(f); - return err; - } - - err = fseek(f, off, SEEK_SET); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_prog -> %d", err); - fclose(f); - return err; - } - - uint8_t dat; - res = fread(&dat, 1, 1, f); - if (res < 1) { - err = -errno; - LFS_TRACE("lfs_emubd_prog -> %d", err); - fclose(f); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_prog -> %d", err); - return err; - } - - // update history and stats - if (block != emu->history.blocks[0]) { - memmove(&emu->history.blocks[1], &emu->history.blocks[0], - sizeof(emu->history) - sizeof(emu->history.blocks[0])); - emu->history.blocks[0] = block; - } - - emu->stats.prog_count += size; - LFS_TRACE("lfs_emubd_prog -> %d", 0); - return 0; -} - -int lfs_emubd_erase(const struct lfs_config *cfg, lfs_block_t block) { - LFS_TRACE("lfs_emubd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); - lfs_emubd_t *emu = cfg->context; - - // Check if erase is valid - LFS_ASSERT(block < cfg->block_count); - - // Erase the block - snprintf(emu->child, LFS_NAME_MAX, "%" PRIx32, block); - struct stat st; - int err = stat(emu->path, &st); - if (err && errno != ENOENT) { - err = -errno; - LFS_TRACE("lfs_emubd_erase -> %d", err); - return err; - } - - if (!err && S_ISREG(st.st_mode) && (S_IWUSR & st.st_mode)) { - err = unlink(emu->path); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_erase -> %d", err); - return err; - } - } - - if (err || (S_ISREG(st.st_mode) && (S_IWUSR & st.st_mode))) { - FILE *f = fopen(emu->path, "w"); - if (!f) { - err = -errno; - LFS_TRACE("lfs_emubd_erase -> %d", err); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_erase -> %d", err); - return err; - } - } - - emu->stats.erase_count += cfg->block_size; - LFS_TRACE("lfs_emubd_erase -> %d", 0); - return 0; -} - -int lfs_emubd_sync(const struct lfs_config *cfg) { - LFS_TRACE("lfs_emubd_sync(%p)", (void*)cfg); - lfs_emubd_t *emu = cfg->context; - - // Just write out info/stats for later lookup - snprintf(emu->child, LFS_NAME_MAX, ".config"); - FILE *f = fopen(emu->path, "w"); - if (!f) { - int err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - return err; - } - - lfs_emubd_tole32(emu); - size_t res = fwrite(&emu->cfg, sizeof(emu->cfg), 1, f); - lfs_emubd_fromle32(emu); - if (res < 1) { - int err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - fclose(f); - return err; - } - - int err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - return err; - } - - snprintf(emu->child, LFS_NAME_MAX, ".stats"); - f = fopen(emu->path, "w"); - if (!f) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - return err; - } - - lfs_emubd_tole32(emu); - res = fwrite(&emu->stats, sizeof(emu->stats), 1, f); - lfs_emubd_fromle32(emu); - if (res < 1) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - fclose(f); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - return err; - } - - snprintf(emu->child, LFS_NAME_MAX, ".history"); - f = fopen(emu->path, "w"); - if (!f) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - return err; - } - - lfs_emubd_tole32(emu); - res = fwrite(&emu->history, sizeof(emu->history), 1, f); - lfs_emubd_fromle32(emu); - if (res < 1) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - fclose(f); - return err; - } - - err = fclose(f); - if (err) { - err = -errno; - LFS_TRACE("lfs_emubd_sync -> %d", err); - return err; - } - - LFS_TRACE("lfs_emubd_sync -> %d", 0); - return 0; -} diff --git a/emubd/lfs_emubd.h b/emubd/lfs_emubd.h deleted file mode 100644 index 0fd78c11..00000000 --- a/emubd/lfs_emubd.h +++ /dev/null @@ -1,79 +0,0 @@ -/* - * Block device emulated on standard files - * - * Copyright (c) 2017, Arm Limited. All rights reserved. - * SPDX-License-Identifier: BSD-3-Clause - */ -#ifndef LFS_EMUBD_H -#define LFS_EMUBD_H - -#include "lfs.h" -#include "lfs_util.h" - -#ifdef __cplusplus -extern "C" -{ -#endif - - -// Config options -#ifndef LFS_EMUBD_ERASE_VALUE -#define LFS_EMUBD_ERASE_VALUE 0x00 -#endif - - -// The emu bd state -typedef struct lfs_emubd { - char *path; - char *child; - - struct { - uint64_t read_count; - uint64_t prog_count; - uint64_t erase_count; - } stats; - - struct { - lfs_block_t blocks[4]; - } history; - - struct { - uint32_t read_size; - uint32_t prog_size; - uint32_t block_size; - uint32_t block_count; - } cfg; -} lfs_emubd_t; - - -// Create a block device using path for the directory to store blocks -int lfs_emubd_create(const struct lfs_config *cfg, const char *path); - -// Clean up memory associated with emu block device -void lfs_emubd_destroy(const struct lfs_config *cfg); - -// Read a block -int lfs_emubd_read(const struct lfs_config *cfg, lfs_block_t block, - lfs_off_t off, void *buffer, lfs_size_t size); - -// Program a block -// -// The block must have previously been erased. -int lfs_emubd_prog(const struct lfs_config *cfg, lfs_block_t block, - lfs_off_t off, const void *buffer, lfs_size_t size); - -// Erase a block -// -// A block must be erased before being programmed. The -// state of an erased block is undefined. -int lfs_emubd_erase(const struct lfs_config *cfg, lfs_block_t block); - -// Sync the block device -int lfs_emubd_sync(const struct lfs_config *cfg); - - -#ifdef __cplusplus -} /* extern "C" */ -#endif - -#endif diff --git a/filebd/lfs_filebd.c b/filebd/lfs_filebd.c new file mode 100644 index 00000000..d7fd89f5 --- /dev/null +++ b/filebd/lfs_filebd.c @@ -0,0 +1,198 @@ +/* + * Block device emulated in a file + * + * Copyright (c) 2017, Arm Limited. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + */ +#include "filebd/lfs_filebd.h" + +#include +#include +#include + +int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, + const struct lfs_filebd_config *filecfg) { + LFS_TRACE("lfs_filebd_createcfg(%p {.context=%p, " + ".read=%p, .prog=%p, .erase=%p, .sync=%p, " + ".read_size=%"PRIu32", .prog_size=%"PRIu32", " + ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " + "\"%s\", " + "%p {.erase_value=%"PRId32"})", + (void*)cfg, cfg->context, + (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, + (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, + cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, + path, (void*)filecfg, filecfg->erase_value); + lfs_filebd_t *bd = cfg->context; + bd->cfg = filecfg; + + // open file + bd->fd = open(path, O_RDWR | O_CREAT, 0666); + if (bd->fd < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_createcfg -> %d", err); + return err; + } + + LFS_TRACE("lfs_filebd_createcfg -> %d", 0); + return 0; +} + +int lfs_filebd_create(const struct lfs_config *cfg, const char *path) { + LFS_TRACE("lfs_filebd_create(%p {.context=%p, " + ".read=%p, .prog=%p, .erase=%p, .sync=%p, " + ".read_size=%"PRIu32", .prog_size=%"PRIu32", " + ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " + "\"%s\")", + (void*)cfg, cfg->context, + (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, + (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, + cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, + path); + static const struct lfs_filebd_config defaults = {.erase_value=-1}; + int err = lfs_filebd_createcfg(cfg, path, &defaults); + LFS_TRACE("lfs_filebd_create -> %d", err); + return err; +} + +void lfs_filebd_destroy(const struct lfs_config *cfg) { + LFS_TRACE("lfs_filebd_destroy(%p)", (void*)cfg); + lfs_filebd_t *bd = cfg->context; + close(bd->fd); + LFS_TRACE("lfs_filebd_destroy -> %s", "void"); +} + +int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size) { + LFS_TRACE("lfs_filebd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + (void*)cfg, block, off, buffer, size); + lfs_filebd_t *bd = cfg->context; + + // check if read is valid + LFS_ASSERT(off % cfg->read_size == 0); + LFS_ASSERT(size % cfg->read_size == 0); + LFS_ASSERT(block < cfg->block_count); + + // zero for reproducability (in case file is truncated) + if (bd->cfg->erase_value != -1) { + memset(buffer, bd->cfg->erase_value, size); + } + + // read + off_t res1 = lseek(bd->fd, + (off_t)block*cfg->block_size + (off_t)off, SEEK_SET); + if (res1 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_read -> %d", err); + return err; + } + + ssize_t res2 = read(bd->fd, buffer, size); + if (res2 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_read -> %d", err); + return err; + } + + LFS_TRACE("lfs_filebd_read -> %d", 0); + return 0; +} + +int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size) { + LFS_TRACE("lfs_filebd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + (void*)cfg, block, off, buffer, size); + lfs_filebd_t *bd = cfg->context; + + // check if write is valid + LFS_ASSERT(off % cfg->prog_size == 0); + LFS_ASSERT(size % cfg->prog_size == 0); + LFS_ASSERT(block < cfg->block_count); + + // check that data was erased? only needed for testing + if (bd->cfg->erase_value != -1) { + off_t res1 = lseek(bd->fd, + (off_t)block*cfg->block_size + (off_t)off, SEEK_SET); + if (res1 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_prog -> %d", err); + return err; + } + + for (lfs_off_t i = 0; i < size; i++) { + uint8_t c; + ssize_t res2 = read(bd->fd, &c, 1); + if (res2 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_prog -> %d", err); + return err; + } + + LFS_ASSERT(c == bd->cfg->erase_value); + } + } + + // program data + off_t res1 = lseek(bd->fd, + (off_t)block*cfg->block_size + (off_t)off, SEEK_SET); + if (res1 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_prog -> %d", err); + return err; + } + + ssize_t res2 = write(bd->fd, buffer, size); + if (res2 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_prog -> %d", err); + return err; + } + + LFS_TRACE("lfs_filebd_prog -> %d", 0); + return 0; +} + +int lfs_filebd_erase(const struct lfs_config *cfg, lfs_block_t block) { + LFS_TRACE("lfs_filebd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); + lfs_filebd_t *bd = cfg->context; + + // check if erase is valid + LFS_ASSERT(block < cfg->block_count); + + // erase, only needed for testing + if (bd->cfg->erase_value != -1) { + off_t res1 = lseek(bd->fd, (off_t)block*cfg->block_size, SEEK_SET); + if (res1 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_erase -> %d", err); + return err; + } + + for (lfs_off_t i = 0; i < cfg->block_size; i++) { + ssize_t res2 = write(bd->fd, &(uint8_t){bd->cfg->erase_value}, 1); + if (res2 < 0) { + int err = -errno; + LFS_TRACE("lfs_filebd_erase -> %d", err); + return err; + } + } + } + + LFS_TRACE("lfs_filebd_erase -> %d", 0); + return 0; +} + +int lfs_filebd_sync(const struct lfs_config *cfg) { + LFS_TRACE("lfs_filebd_sync(%p)", (void*)cfg); + // file sync + lfs_filebd_t *bd = cfg->context; + int err = fsync(bd->fd); + if (err) { + err = -errno; + LFS_TRACE("lfs_filebd_sync -> %d", 0); + return err; + } + + LFS_TRACE("lfs_filebd_sync -> %d", 0); + return 0; +} diff --git a/filebd/lfs_filebd.h b/filebd/lfs_filebd.h new file mode 100644 index 00000000..ae288582 --- /dev/null +++ b/filebd/lfs_filebd.h @@ -0,0 +1,64 @@ +/* + * Block device emulated in a file + * + * Copyright (c) 2017, Arm Limited. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + */ +#ifndef LFS_FILEBD_H +#define LFS_FILEBD_H + +#include "lfs.h" +#include "lfs_util.h" + +#ifdef __cplusplus +extern "C" +{ +#endif + +// filebd config (optional) +struct lfs_filebd_config { + // 8-bit erase value to simulate erasing with. -1 indicates no erase + // occurs, which is still a valid block device + int32_t erase_value; +}; + +// filebd state +typedef struct lfs_filebd { + int fd; + const struct lfs_filebd_config *cfg; +} lfs_filebd_t; + + +// Create a file block device using the geometry in lfs_config +int lfs_filebd_create(const struct lfs_config *cfg, const char *path); +int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, + const struct lfs_filebd_config *ramcfg); + +// Clean up memory associated with block device +void lfs_filebd_destroy(const struct lfs_config *cfg); + +// Read a block +int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size); + +// Program a block +// +// The block must have previously been erased. +int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size); + +// Erase a block +// +// A block must be erased before being programmed. The +// state of an erased block is undefined. +int lfs_filebd_erase(const struct lfs_config *cfg, lfs_block_t block); + +// Sync the block device +int lfs_filebd_sync(const struct lfs_config *cfg); + + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif diff --git a/rambd/lfs_rambd.c b/rambd/lfs_rambd.c new file mode 100644 index 00000000..5bf8cc13 --- /dev/null +++ b/rambd/lfs_rambd.c @@ -0,0 +1,137 @@ +/* + * Block device emulated in RAM + * + * Copyright (c) 2017, Arm Limited. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + */ +#include "rambd/lfs_rambd.h" + +int lfs_rambd_createcfg(const struct lfs_config *cfg, + const struct lfs_rambd_config *ramcfg) { + LFS_TRACE("lfs_rambd_createcfg(%p {.context=%p, " + ".read=%p, .prog=%p, .erase=%p, .sync=%p, " + ".read_size=%"PRIu32", .prog_size=%"PRIu32", " + ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " + "%p {.erase_value=%"PRId32", .buffer=%p})", + (void*)cfg, cfg->context, + (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, + (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, + cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, + (void*)ramcfg, ramcfg->erase_value, ramcfg->buffer); + lfs_rambd_t *bd = cfg->context; + bd->cfg = ramcfg; + + // allocate buffer? + if (bd->cfg->buffer) { + bd->buffer = bd->cfg->buffer; + } else { + bd->buffer = lfs_malloc(cfg->block_size * cfg->block_count); + if (!bd->buffer) { + LFS_TRACE("lfs_rambd_createcfg -> %d", LFS_ERR_NOMEM); + return LFS_ERR_NOMEM; + } + } + + // zero for reproducability? + if (bd->cfg->erase_value != -1) { + memset(bd->buffer, bd->cfg->erase_value, + cfg->block_size * cfg->block_count); + } + + LFS_TRACE("lfs_rambd_createcfg -> %d", 0); + return 0; +} + +int lfs_rambd_create(const struct lfs_config *cfg) { + LFS_TRACE("lfs_rambd_create(%p {.context=%p, " + ".read=%p, .prog=%p, .erase=%p, .sync=%p, " + ".read_size=%"PRIu32", .prog_size=%"PRIu32", " + ".block_size=%"PRIu32", .block_count=%"PRIu32"})", + (void*)cfg, cfg->context, + (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, + (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, + cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count); + static const struct lfs_rambd_config defaults = {.erase_value=-1}; + int err = lfs_rambd_createcfg(cfg, &defaults); + LFS_TRACE("lfs_rambd_create -> %d", err); + return err; +} + +void lfs_rambd_destroy(const struct lfs_config *cfg) { + LFS_TRACE("lfs_rambd_destroy(%p)", (void*)cfg); + // clean up memory + lfs_rambd_t *bd = cfg->context; + if (!bd->cfg->buffer) { + lfs_free(bd->buffer); + } + LFS_TRACE("lfs_rambd_destroy -> %s", "void"); +} + +int lfs_rambd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size) { + LFS_TRACE("lfs_rambd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + (void*)cfg, block, off, buffer, size); + lfs_rambd_t *bd = cfg->context; + + // check if read is valid + LFS_ASSERT(off % cfg->read_size == 0); + LFS_ASSERT(size % cfg->read_size == 0); + LFS_ASSERT(block < cfg->block_count); + + // read data + memcpy(buffer, &bd->buffer[block*cfg->block_size + off], size); + + LFS_TRACE("lfs_rambd_read -> %d", 0); + return 0; +} + +int lfs_rambd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size) { + LFS_TRACE("lfs_rambd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + (void*)cfg, block, off, buffer, size); + lfs_rambd_t *bd = cfg->context; + + // check if write is valid + LFS_ASSERT(off % cfg->prog_size == 0); + LFS_ASSERT(size % cfg->prog_size == 0); + LFS_ASSERT(block < cfg->block_count); + + // check that data was erased? only needed for testing + if (bd->cfg->erase_value != -1) { + for (lfs_off_t i = 0; i < size; i++) { + LFS_ASSERT(bd->buffer[block*cfg->block_size + off + i] == + bd->cfg->erase_value); + } + } + + // program data + memcpy(&bd->buffer[block*cfg->block_size + off], buffer, size); + + LFS_TRACE("lfs_rambd_prog -> %d", 0); + return 0; +} + +int lfs_rambd_erase(const struct lfs_config *cfg, lfs_block_t block) { + LFS_TRACE("lfs_rambd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); + lfs_rambd_t *bd = cfg->context; + + // check if erase is valid + LFS_ASSERT(block < cfg->block_count); + + // erase, only needed for testing + if (bd->cfg->erase_value != -1) { + memset(&bd->buffer[block*cfg->block_size], + bd->cfg->erase_value, cfg->block_size); + } + + LFS_TRACE("lfs_rambd_erase -> %d", 0); + return 0; +} + +int lfs_rambd_sync(const struct lfs_config *cfg) { + LFS_TRACE("lfs_rambd_sync(%p)", (void*)cfg); + // sync does nothing because we aren't backed by anything real + (void)cfg; + LFS_TRACE("lfs_rambd_sync -> %d", 0); + return 0; +} diff --git a/rambd/lfs_rambd.h b/rambd/lfs_rambd.h new file mode 100644 index 00000000..a4b95151 --- /dev/null +++ b/rambd/lfs_rambd.h @@ -0,0 +1,67 @@ +/* + * Block device emulated in RAM + * + * Copyright (c) 2017, Arm Limited. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + */ +#ifndef LFS_RAMBD_H +#define LFS_RAMBD_H + +#include "lfs.h" +#include "lfs_util.h" + +#ifdef __cplusplus +extern "C" +{ +#endif + +// rambd config (optional) +struct lfs_rambd_config { + // 8-bit erase value to simulate erasing with. -1 indicates no erase + // occurs, which is still a valid block device + int32_t erase_value; + + // Optional statically allocated buffer for the block device. + void *buffer; +}; + +// rambd state +typedef struct lfs_rambd { + uint8_t *buffer; + const struct lfs_rambd_config *cfg; +} lfs_rambd_t; + + +// Create a RAM block device using the geometry in lfs_config +int lfs_rambd_create(const struct lfs_config *cfg); +int lfs_rambd_createcfg(const struct lfs_config *cfg, + const struct lfs_rambd_config *ramcfg); + +// Clean up memory associated with block device +void lfs_rambd_destroy(const struct lfs_config *cfg); + +// Read a block +int lfs_rambd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size); + +// Program a block +// +// The block must have previously been erased. +int lfs_rambd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size); + +// Erase a block +// +// A block must be erased before being programmed. The +// state of an erased block is undefined. +int lfs_rambd_erase(const struct lfs_config *cfg, lfs_block_t block); + +// Sync the block device +int lfs_rambd_sync(const struct lfs_config *cfg); + + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif diff --git a/scripts/test_.py b/scripts/test_.py index 5a481e40..27002d9a 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -15,7 +15,6 @@ import base64 import sys import copy -import shutil import shlex TESTDIR = 'tests_' @@ -40,8 +39,10 @@ GLOBALS = """ //////////////// AUTOGENERATED TEST //////////////// #include "lfs.h" -#include "emubd/lfs_emubd.h" +#include "filebd/lfs_filebd.h" +#include "rambd/lfs_rambd.h" #include +const char *LFS_DISK = NULL; """ DEFINES = { "LFS_READ_SIZE": 16, @@ -51,23 +52,25 @@ "LFS_BLOCK_CYCLES": 1024, "LFS_CACHE_SIZE": "(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)", "LFS_LOOKAHEAD_SIZE": 16, + "LFS_ERASE_VALUE": 0xff, } PROLOGUE = """ // prologue __attribute__((unused)) lfs_t lfs; - __attribute__((unused)) lfs_emubd_t bd; + __attribute__((unused)) lfs_filebd_t filebd; + __attribute__((unused)) lfs_rambd_t rambd; __attribute__((unused)) lfs_file_t file; __attribute__((unused)) lfs_dir_t dir; __attribute__((unused)) struct lfs_info info; __attribute__((unused)) uint8_t buffer[1024]; __attribute__((unused)) char path[1024]; - + __attribute__((unused)) const struct lfs_config cfg = { - .context = &bd, - .read = &lfs_emubd_read, - .prog = &lfs_emubd_prog, - .erase = &lfs_emubd_erase, - .sync = &lfs_emubd_sync, + .context = LFS_DISK ? (void*)&filebd : (void*)&rambd, + .read = LFS_DISK ? &lfs_filebd_read : &lfs_rambd_read, + .prog = LFS_DISK ? &lfs_filebd_prog : &lfs_rambd_prog, + .erase = LFS_DISK ? &lfs_filebd_erase : &lfs_rambd_erase, + .sync = LFS_DISK ? &lfs_filebd_sync : &lfs_rambd_sync, .read_size = LFS_READ_SIZE, .prog_size = LFS_PROG_SIZE, @@ -78,11 +81,26 @@ .lookahead_size = LFS_LOOKAHEAD_SIZE, }; - lfs_emubd_create(&cfg, "blocks"); + __attribute__((unused)) const struct lfs_filebd_config filecfg = { + .erase_value = LFS_ERASE_VALUE, + }; + __attribute__((unused)) const struct lfs_rambd_config ramcfg = { + .erase_value = LFS_ERASE_VALUE, + }; + + if (LFS_DISK) { + lfs_filebd_createcfg(&cfg, LFS_DISK, &filecfg); + } else { + lfs_rambd_createcfg(&cfg, &ramcfg); + } """ EPILOGUE = """ // epilogue - lfs_emubd_destroy(&cfg); + if (LFS_DISK) { + lfs_filebd_destroy(&cfg); + } else { + lfs_rambd_destroy(&cfg); + } """ PASS = '\033[32m✓\033[0m' FAIL = '\033[31m✗\033[0m' @@ -133,7 +151,8 @@ def build(self, f, **_): f.write(') {\n') for k, v in sorted(self.defines.items()): - f.write(4*' '+'#define %s %s\n' % (k, v)) + if k not in self.suite.defines: + f.write(4*' '+'#define %s %s\n' % (k, v)) f.write(PROLOGUE) f.write('\n') @@ -148,27 +167,34 @@ def build(self, f, **_): f.write('\n') for k, v in sorted(self.defines.items()): - f.write(4*' '+'#undef %s\n' % k) + if k not in self.suite.defines: + f.write(4*' '+'#undef %s\n' % k) f.write('}\n') def test(self, exec=[], persist=False, gdb=False, failure=None, **args): - # clear disk first - if not persist: - shutil.rmtree('blocks', True) - # build command cmd = exec + ['./%s.test' % self.suite.path, repr(self.caseno), repr(self.permno)] + if persist: + cmd.append(self.suite.path + '.test.disk') # failed? drop into debugger? if gdb and failure: - cmd = (['gdb', '-ex', 'r' - ] + (['-ex', 'up'] if failure.assert_ else []) + [ - '--args'] + cmd) + ncmd = ['gdb'] + if gdb == 'assert': + ncmd.extend(['-ex', 'r']) + if failure.assert_: + ncmd.extend(['-ex', 'up']) + elif gdb == 'start': + ncmd.extend([ + '-ex', 'b %s:%d' % (self.suite.path, self.lineno), + '-ex', 'r']) + ncmd.extend(['--args'] + cmd) + if args.get('verbose', False): - print(' '.join(shlex.quote(c) for c in cmd)) - sys.exit(sp.call(cmd)) + print(' '.join(shlex.quote(c) for c in ncmd)) + sys.exit(sp.call(ncmd)) # run test case! stdout = [] @@ -231,22 +257,27 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): if not self.reentrant: return - for cycles in it.count(1): - npersist = persist or cycles > 1 + # clear disk first? + if not persist: + try: + os.remove(self.suite.path + '.test.disk') + except FileNotFoundError: + pass + for cycles in it.count(1): # exact cycle we should drop into debugger? if gdb and failure and failure.cycleno == cycles: - return super().test(exec=exec, persist=npersist, + return super().test(exec=exec, persist=True, gdb=gdb, failure=failure, **args) - # run tests, but kill the program after lfs_emubd_prog/erase has + # run tests, but kill the program after prog/erase has # been hit n cycles. We exit with a special return code if the # program has not finished, since this isn't a test failure. nexec = exec + [ 'gdb', '-batch-silent', '-ex', 'handle all nostop', - '-ex', 'b lfs_emubd_prog', - '-ex', 'b lfs_emubd_erase', + '-ex', 'b lfs_filebd_prog', + '-ex', 'b lfs_filebd_erase', '-ex', 'r', ] + cycles*['-ex', 'c'] + [ '-ex', 'q ' @@ -255,7 +286,7 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): '33', '--args'] try: - return super().test(exec=nexec, persist=npersist, **args) + return super().test(exec=nexec, persist=True, **args) except TestFailure as nfailure: if nfailure.returncode == 33: continue @@ -370,10 +401,11 @@ def build(self, **args): f.write('\n') f.write('int main(int argc, char **argv) {\n') - f.write(4*' '+'int case_ = (argc == 3) ? atoi(argv[1]) : 0;\n') - f.write(4*' '+'int perm = (argc == 3) ? atoi(argv[2]) : 0;\n') + f.write(4*' '+'int case_ = (argc >= 3) ? atoi(argv[1]) : 0;\n') + f.write(4*' '+'int perm = (argc >= 3) ? atoi(argv[2]) : 0;\n') + f.write(4*' '+'LFS_DISK = (argc >= 4) ? argv[3] : NULL;\n') for perm in self.perms: - f.write(4*' '+'if (argc != 3 || ' + f.write(4*' '+'if (argc < 3 || ' '(case_ == %d && perm == %d)) { ' % ( perm.caseno, perm.permno)) f.write('test_case%d(' % perm.caseno) @@ -590,8 +622,9 @@ def main(**args): help="Run all tests instead of stopping on first error. Useful for CI.") parser.add_argument('-p', '--persist', action='store_true', help="Don't reset the tests disk before each test.") - parser.add_argument('-g', '--gdb', action='store_true', - help="Drop into gdb on failure.") + parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], + nargs='?', const='assert', + help="Drop into gdb on test failure.") parser.add_argument('--valgrind', action='store_true', help="Run non-leaky tests under valgrind to check for memory leaks.") parser.add_argument('--reentrant', action='store_true', From 1d2688a7714d2d0d560dc47acc87b19ed03dba9f Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sat, 11 Jan 2020 15:58:17 -0600 Subject: [PATCH 05/41] Migrated test_files, test_dirs, test_format suites to new framework Also some tweaks to test_.py to capture Makefile warnings and print test locations a bit better. --- scripts/test_.py | 54 +++- tests_/test_dirs.toml | 580 +++++++++++++++++++++++++++++++++++++--- tests_/test_files.toml | 486 +++++++++++++++++++++++++++++++++ tests_/test_format.toml | 105 ++++++++ 4 files changed, 1172 insertions(+), 53 deletions(-) create mode 100644 tests_/test_files.toml create mode 100644 tests_/test_format.toml diff --git a/scripts/test_.py b/scripts/test_.py index 27002d9a..2f016e43 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -62,8 +62,10 @@ __attribute__((unused)) lfs_file_t file; __attribute__((unused)) lfs_dir_t dir; __attribute__((unused)) struct lfs_info info; - __attribute__((unused)) uint8_t buffer[1024]; __attribute__((unused)) char path[1024]; + __attribute__((unused)) uint8_t buffer[1024]; + __attribute__((unused)) lfs_size_t size; + __attribute__((unused)) int err; __attribute__((unused)) const struct lfs_config cfg = { .context = LFS_DISK ? (void*)&filebd : (void*)&rambd, @@ -124,9 +126,11 @@ def __init__(self, config, suite=None, caseno=None, lineno=None, **_): def __str__(self): if hasattr(self, 'permno'): - return '%s[%d,%d]' % (self.suite.name, self.caseno, self.permno) + return '%s[%d,%d]' % ( + self.suite.name, self.caseno, self.permno) else: - return '%s[%d]' % (self.suite.name, self.caseno) + return '%s[%d]' % ( + self.suite.name, self.caseno) def permute(self, defines, permno=None, **_): ncase = copy.copy(self) @@ -211,7 +215,10 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): if args.get('verbose', False): sys.stdout.write(line) # intercept asserts - m = re.match('^([^:]+):([0-9]+):(assert): (.*)$', line) + m = re.match( + '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' + .format('(?:\033\[[\d;]*.| )*', 'assert'), + line) if m and assert_ is None: try: with open(m.group(1)) as f: @@ -221,7 +228,7 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): 'path': m.group(1), 'line': line, 'lineno': lineno, - 'message': m.group(4)} + 'message': m.group(3)} except: pass proc.wait() @@ -385,7 +392,7 @@ def permute(self, defines={}, **args): self.defines = {} for k, v in self.perms[0].defines.items(): - if all(perm.defines[k] == v for perm in self.perms): + if all(perm.defines.get(k, None) == v for perm in self.perms): self.defines[k] = v return self.perms @@ -401,7 +408,7 @@ def build(self, **args): f.write('\n') f.write('int main(int argc, char **argv) {\n') - f.write(4*' '+'int case_ = (argc >= 3) ? atoi(argv[1]) : 0;\n') + f.write(4*' '+'int case_ = (argc >= 2) ? atoi(argv[1]) : 0;\n') f.write(4*' '+'int perm = (argc >= 3) ? atoi(argv[2]) : 0;\n') f.write(4*' '+'LFS_DISK = (argc >= 4) ? argv[3] : NULL;\n') for perm in self.perms: @@ -544,6 +551,23 @@ def main(**args): stdout.append(line) if args.get('verbose', False): sys.stdout.write(line) + # intercept warnings + m = re.match( + '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' + .format('(?:\033\[[\d;]*.| )*', 'warning'), + line) + if m and not args.get('verbose', False): + try: + with open(m.group(1)) as f: + lineno = int(m.group(2)) + line = next(it.islice(f, lineno-1, None)).strip('\n') + sys.stdout.write( + "\033[01m{path}:{lineno}:\033[01;35mwarning:\033[m " + "{message}\n{line}\n\n".format( + path=m.group(1), line=line, lineno=lineno, + message=m.group(3))) + except: + pass proc.wait() if proc.returncode != 0: @@ -587,12 +611,20 @@ def main(**args): if perm.result == PASS: passed += 1 else: - sys.stdout.write("--- %s ---\n" % perm) - if perm.result.assert_: - for line in perm.result.stdout[:-1]: + #sys.stdout.write("--- %s ---\n" % perm) + sys.stdout.write( + "\033[01m{path}:{lineno}:\033[01;31mfailure:\033[m " + "{perm} failed with {returncode}\n".format( + perm=perm, path=perm.suite.path, lineno=perm.lineno-2, + returncode=perm.result.returncode or 0)) + if perm.result.stdout: + for line in (perm.result.stdout + if not perm.result.assert_ + else perm.result.stdout[:-1]): sys.stdout.write(line) + if perm.result.assert_: sys.stdout.write( - "\033[97m{path}:{lineno}:\033[91massert:\033[0m " + "\033[01m{path}:{lineno}:\033[01;31massert:\033[m " "{message}\n{line}\n".format( **perm.result.assert_)) else: diff --git a/tests_/test_dirs.toml b/tests_/test_dirs.toml index 41789d59..e3e04533 100644 --- a/tests_/test_dirs.toml +++ b/tests_/test_dirs.toml @@ -1,28 +1,5 @@ -[[case]] # format -code = """ - lfs_format(&lfs, &cfg) => 0; -""" - -[[case]] # mount/unmount -code = """ - lfs_format(&lfs, &cfg) => 0; - lfs_mount(&lfs, &cfg) => 0; - lfs_unmount(&lfs) => 0; -""" - -[[case]] # reentrant format -code = """ - int err = lfs_mount(&lfs, &cfg); - if (err) { - lfs_format(&lfs, &cfg) => 0; - lfs_mount(&lfs, &cfg) => 0; - } - lfs_unmount(&lfs) => 0; -""" -reentrant = true - [[case]] # root -code = """ +code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; lfs_dir_open(&lfs, &dir, "/") => 0; @@ -35,10 +12,10 @@ code = """ lfs_dir_read(&lfs, &dir, &info) => 0; lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; -""" +''' -[[case]] # directory creation -code = """ +[[case]] # many directory creation +code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -65,11 +42,11 @@ code = """ lfs_dir_read(&lfs, &dir, &info) => 0; lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; -""" +''' define.N = 'range(0, 100, 3)' -[[case]] # directory removal -code = """ +[[case]] # many directory removal +code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -115,19 +92,159 @@ code = """ lfs_dir_read(&lfs, &dir, &info) => 0; lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; -""" +''' +define.N = 'range(3, 100, 11)' + +[[case]] # many directory rename +code = ''' + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "test%03d", i); + lfs_mkdir(&lfs, path) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "test%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + char oldpath[128]; + char newpath[128]; + sprintf(oldpath, "test%03d", i); + sprintf(newpath, "tedd%03d", i); + lfs_rename(&lfs, oldpath, newpath) => 0; + } + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "tedd%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); +''' define.N = 'range(3, 100, 11)' +[[case]] # reentrant many directory creation/rename/removal +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + for (int i = 0; i < N; i++) { + sprintf(path, "hi%03d", i); + err = lfs_mkdir(&lfs, path); + assert(err == 0 || err == LFS_ERR_EXIST); + } + + for (int i = 0; i < N; i++) { + sprintf(path, "hello%03d", i); + err = lfs_remove(&lfs, path); + assert(err == 0 || err == LFS_ERR_NOENT); + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "hi%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int i = 0; i < N; i++) { + char oldpath[128]; + char newpath[128]; + sprintf(oldpath, "hi%03d", i); + sprintf(newpath, "hello%03d", i); + // YES this can overwrite an existing newpath + lfs_rename(&lfs, oldpath, newpath) => 0; + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "hello%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int i = 0; i < N; i++) { + sprintf(path, "hello%03d", i); + lfs_remove(&lfs, path) => 0; + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +''' +define.N = [5, 25] +reentrant = true + [[case]] # file creation -code = """ +code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; for (int i = 0; i < N; i++) { sprintf(path, "file%03d", i); - lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; lfs_file_close(&lfs, &file) => 0; - } + } lfs_unmount(&lfs) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -147,17 +264,18 @@ code = """ lfs_dir_read(&lfs, &dir, &info) => 0; lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs); -""" +''' define.N = 'range(3, 100, 11)' [[case]] # file removal -code = """ +code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; for (int i = 0; i < N; i++) { sprintf(path, "removeme%03d", i); - lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; lfs_file_close(&lfs, &file) => 0; } lfs_unmount(&lfs) => 0; @@ -198,23 +316,401 @@ code = """ lfs_dir_read(&lfs, &dir, &info) => 0; lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; -""" +''' +define.N = 'range(0, 100, 3)' + +[[case]] # file rename +code = ''' + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "test%03d", i); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "test%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + char oldpath[128]; + char newpath[128]; + sprintf(oldpath, "test%03d", i); + sprintf(newpath, "tedd%03d", i); + lfs_rename(&lfs, oldpath, newpath) => 0; + } + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "tedd%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); +''' define.N = 'range(0, 100, 3)' -[[case]] # error cases -code = """ +[[case]] # reentrant file creation/rename/removal +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + for (int i = 0; i < N; i++) { + sprintf(path, "hi%03d", i); + lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_close(&lfs, &file) => 0; + } + + for (int i = 0; i < N; i++) { + sprintf(path, "hello%03d", i); + err = lfs_remove(&lfs, path); + assert(err == 0 || err == LFS_ERR_NOENT); + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "hi%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int i = 0; i < N; i++) { + char oldpath[128]; + char newpath[128]; + sprintf(oldpath, "hi%03d", i); + sprintf(newpath, "hello%03d", i); + // YES this can overwrite an existing newpath + lfs_rename(&lfs, oldpath, newpath) => 0; + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "hello%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int i = 0; i < N; i++) { + sprintf(path, "hello%03d", i); + lfs_remove(&lfs, path) => 0; + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +''' +define.N = [5, 25] +reentrant = true + +[[case]] # nested directories +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "potato") => 0; + lfs_file_open(&lfs, &file, "burito", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "potato/baked") => 0; + lfs_mkdir(&lfs, "potato/sweet") => 0; + lfs_mkdir(&lfs, "potato/fried") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "potato") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "baked") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "fried") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "sweet") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + + // try removing? + lfs_mount(&lfs, &cfg) => 0; + lfs_remove(&lfs, "potato") => LFS_ERR_NOTEMPTY; + lfs_unmount(&lfs) => 0; + + // try renaming? + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "potato", "coldpotato") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "coldpotato", "warmpotato") => 0; + lfs_rename(&lfs, "warmpotato", "hotpotato") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_remove(&lfs, "potato") => LFS_ERR_NOENT; + lfs_remove(&lfs, "coldpotato") => LFS_ERR_NOENT; + lfs_remove(&lfs, "warmpotato") => LFS_ERR_NOENT; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_unmount(&lfs) => 0; + + // try cross-directory renaming + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "coldpotato") => 0; + lfs_rename(&lfs, "hotpotato/baked", "coldpotato/baked") => 0; + lfs_rename(&lfs, "coldpotato", "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "coldpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_rename(&lfs, "hotpotato/fried", "coldpotato/fried") => 0; + lfs_rename(&lfs, "coldpotato", "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "coldpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_rename(&lfs, "hotpotato/sweet", "coldpotato/sweet") => 0; + lfs_rename(&lfs, "coldpotato", "hotpotato") => 0; + lfs_remove(&lfs, "coldpotato") => LFS_ERR_NOENT; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "hotpotato") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "baked") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "fried") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "sweet") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + + // final remove + lfs_mount(&lfs, &cfg) => 0; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "hotpotato/baked") => 0; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "hotpotato/fried") => 0; + lfs_remove(&lfs, "hotpotato") => LFS_ERR_NOTEMPTY; + lfs_remove(&lfs, "hotpotato/sweet") => 0; + lfs_remove(&lfs, "hotpotato") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + info.type => LFS_TYPE_DIR; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "burito") == 0); + info.type => LFS_TYPE_REG; + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # recursive remove +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "prickly-pear") => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "prickly-pear/cactus%03d", i); + lfs_mkdir(&lfs, path) => 0; + } + lfs_dir_open(&lfs, &dir, "prickly-pear") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "cactus%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs); + + lfs_mount(&lfs, &cfg) => 0; + lfs_remove(&lfs, "prickly-pear") => LFS_ERR_NOTEMPTY; + + lfs_dir_open(&lfs, &dir, "prickly-pear") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + for (int i = 0; i < N; i++) { + sprintf(path, "cactus%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, path) == 0); + sprintf(path, "prickly-pear/%s", info.name); + lfs_remove(&lfs, path) => 0; + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_remove(&lfs, "prickly-pear") => 0; + lfs_remove(&lfs, "prickly-pear") => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_remove(&lfs, "prickly-pear") => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' +define.N = [10, 100] + +[[case]] # other error cases +code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; lfs_mkdir(&lfs, "potato") => 0; - lfs_file_open(&lfs, &file, "burito", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_open(&lfs, &file, "burito", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "potato") => LFS_ERR_EXIST; + lfs_mkdir(&lfs, "burito") => LFS_ERR_EXIST; + lfs_file_open(&lfs, &file, "burito", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => LFS_ERR_EXIST; + lfs_file_open(&lfs, &file, "potato", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => LFS_ERR_EXIST; lfs_dir_open(&lfs, &dir, "tomato") => LFS_ERR_NOENT; lfs_dir_open(&lfs, &dir, "burito") => LFS_ERR_NOTDIR; lfs_file_open(&lfs, &file, "tomato", LFS_O_RDONLY) => LFS_ERR_NOENT; lfs_file_open(&lfs, &file, "potato", LFS_O_RDONLY) => LFS_ERR_ISDIR; + lfs_file_open(&lfs, &file, "tomato", LFS_O_WRONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "potato", LFS_O_WRONLY) => LFS_ERR_ISDIR; + lfs_file_open(&lfs, &file, "potato", + LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_ISDIR; + + lfs_mkdir(&lfs, "/") => LFS_ERR_EXIST; + lfs_file_open(&lfs, &file, "/", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => LFS_ERR_EXIST; + lfs_file_open(&lfs, &file, "/", LFS_O_RDONLY) => LFS_ERR_ISDIR; + lfs_file_open(&lfs, &file, "/", LFS_O_WRONLY) => LFS_ERR_ISDIR; + lfs_file_open(&lfs, &file, "/", + LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_ISDIR; + + // check that errors did not corrupt directory + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, "burito") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "potato") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_unmount(&lfs) => 0; + + // or on disk + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, ".") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "..") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_REG); + assert(strcmp(info.name, "burito") == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(info.type == LFS_TYPE_DIR); + assert(strcmp(info.name, "potato") == 0); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; -""" +''' diff --git a/tests_/test_files.toml b/tests_/test_files.toml new file mode 100644 index 00000000..ce161786 --- /dev/null +++ b/tests_/test_files.toml @@ -0,0 +1,486 @@ + +[[case]] # simple file test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "hello", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + size = strlen("Hello World!")+1; + strcpy((char*)buffer, "Hello World!"); + lfs_file_write(&lfs, &file, buffer, size) => size; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(strcmp((char*)buffer, "Hello World!") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # larger files +code = ''' + lfs_format(&lfs, &cfg) => 0; + + // write + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + srand(1); + for (lfs_size_t i = 0; i < SIZE; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE; + srand(1); + for (lfs_size_t i = 0; i < SIZE; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE = [32, 8192, 262144, 0, 7, 8193] +define.CHUNKSIZE = [31, 16, 33, 1, 1023] + +[[case]] # rewriting files +code = ''' + lfs_format(&lfs, &cfg) => 0; + + // write + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE1; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // rewrite + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_WRONLY) => 0; + srand(2); + for (lfs_size_t i = 0; i < SIZE2; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE2-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => lfs_max(SIZE1, SIZE2); + srand(2); + for (lfs_size_t i = 0; i < SIZE2; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE2-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + if (SIZE1 > SIZE2) { + srand(1); + for (lfs_size_t b = 0; b < SIZE2; b++) { + rand(); + } + for (lfs_size_t i = SIZE2; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] +define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] +define.CHUNKSIZE = [31, 16, 1, 1025] + +[[case]] # appending files +code = ''' + lfs_format(&lfs, &cfg) => 0; + + // write + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE1; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // append + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_WRONLY | LFS_O_APPEND) => 0; + srand(2); + for (lfs_size_t i = 0; i < SIZE2; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE2-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE1 + SIZE2; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + srand(2); + for (lfs_size_t i = 0; i < SIZE2; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE2-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] +define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] +define.CHUNKSIZE = [31, 16, 1, 1025] + +[[case]] # truncating files +code = ''' + lfs_format(&lfs, &cfg) => 0; + + // write + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE1; + srand(1); + for (lfs_size_t i = 0; i < SIZE1; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE1-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // truncate + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_WRONLY | LFS_O_TRUNC) => 0; + srand(2); + for (lfs_size_t i = 0; i < SIZE2; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE2-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // read + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE2; + srand(2); + for (lfs_size_t i = 0; i < SIZE2; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE2-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] +define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] +define.CHUNKSIZE = [31, 16, 1, 1025] + +[[case]] # reentrant file writing +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + err = lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY); + assert(err == LFS_ERR_NOENT || err == 0); + if (err == 0) { + // can only be 0 (new file) or full size + size = lfs_file_size(&lfs, &file); + assert(size == 0 || size == SIZE); + lfs_file_close(&lfs, &file) => 0; + } + + // write + lfs_file_open(&lfs, &file, "avacado", LFS_O_WRONLY | LFS_O_CREAT) => 0; + srand(1); + for (lfs_size_t i = 0; i < SIZE; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + } + lfs_file_close(&lfs, &file) => 0; + + // read + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE; + srand(1); + for (lfs_size_t i = 0; i < SIZE; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE = [32, 0, 7, 2049] +define.CHUNKSIZE = [31, 16, 65] +reentrant = true + +[[case]] # reentrant file writing with syncs +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + err = lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY); + assert(err == LFS_ERR_NOENT || err == 0); + if (err == 0) { + // with syncs we could be any size, but it at least must be valid data + size = lfs_file_size(&lfs, &file); + assert(size <= SIZE); + srand(1); + for (lfs_size_t i = 0; i < size; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, size-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_close(&lfs, &file) => 0; + } + + // write + lfs_file_open(&lfs, &file, "avacado", + LFS_O_WRONLY | LFS_O_CREAT | MODE) => 0; + size = lfs_file_size(&lfs, &file); + assert(size <= SIZE); + srand(1); + lfs_size_t skip = (MODE == LFS_O_APPEND) ? size : 0; + for (lfs_size_t b = 0; b < skip; b++) { + rand(); + } + for (lfs_size_t i = skip; i < SIZE; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE-i); + for (lfs_size_t b = 0; b < chunk; b++) { + buffer[b] = rand() & 0xff; + } + lfs_file_write(&lfs, &file, buffer, chunk) => chunk; + lfs_file_sync(&lfs, &file) => 0; + } + lfs_file_close(&lfs, &file) => 0; + + // read + lfs_file_open(&lfs, &file, "avacado", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => SIZE; + srand(1); + for (lfs_size_t i = 0; i < SIZE; i += CHUNKSIZE) { + lfs_size_t chunk = lfs_min(CHUNKSIZE, SIZE-i); + lfs_file_read(&lfs, &file, buffer, chunk) => chunk; + for (lfs_size_t b = 0; b < chunk; b++) { + assert(buffer[b] == (rand() & 0xff)); + } + } + lfs_file_read(&lfs, &file, buffer, CHUNKSIZE) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define = [ + # append (O(n)) + {MODE='LFS_O_APPEND', SIZE=[32, 0, 7, 2049], CHUNKSIZE=[31, 16, 65]}, + # truncate (O(n^2)) + {MODE='LFS_O_TRUNC', SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]}, + # rewrite (O(n^2)) + {MODE=0, SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]}, +] +reentrant = true + +[[case]] # many files +code = ''' + lfs_format(&lfs, &cfg) => 0; + // create N files of 7 bytes + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "file_%03d", i); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + char wbuffer[1024]; + size = 7; + snprintf(wbuffer, size, "Hi %03d", i); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_close(&lfs, &file) => 0; + + char rbuffer[1024]; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + assert(strcmp(rbuffer, wbuffer) == 0); + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; +''' +define.N = 300 + +[[case]] # many files with power cycle +code = ''' + lfs_format(&lfs, &cfg) => 0; + // create N files of 7 bytes + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + sprintf(path, "file_%03d", i); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + char wbuffer[1024]; + size = 7; + snprintf(wbuffer, size, "Hi %03d", i); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + char rbuffer[1024]; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + assert(strcmp(rbuffer, wbuffer) == 0); + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; +''' +define.N = 300 + +[[case]] # many files with power loss +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + // create N files of 7 bytes + for (int i = 0; i < N; i++) { + sprintf(path, "file_%03d", i); + err = lfs_file_open(&lfs, &file, path, LFS_O_WRONLY | LFS_O_CREAT); + char wbuffer[1024]; + size = 7; + snprintf(wbuffer, size, "Hi %03d", i); + if ((lfs_size_t)lfs_file_size(&lfs, &file) != size) { + lfs_file_write(&lfs, &file, wbuffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + char rbuffer[1024]; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + assert(strcmp(rbuffer, wbuffer) == 0); + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; +''' +define.N = 300 +reentrant = true diff --git a/tests_/test_format.toml b/tests_/test_format.toml new file mode 100644 index 00000000..c52db2a4 --- /dev/null +++ b/tests_/test_format.toml @@ -0,0 +1,105 @@ +[[case]] # simple formatting test +code = ''' + lfs_format(&lfs, &cfg) => 0; +''' + +[[case]] # mount/unmount +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # reentrant format +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + lfs_unmount(&lfs) => 0; +''' +reentrant = true + +[[case]] # invalid mount +code = ''' + lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; +''' + +# TODO invalid superblock? (corrupt 1, 0) + +[[case]] # expanding superblock +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + lfs_mkdir(&lfs, "dummy") => 0; + lfs_stat(&lfs, "dummy", &info) => 0; + assert(strcmp(info.name, "dummy") == 0); + lfs_remove(&lfs, "dummy") => 0; + } + lfs_unmount(&lfs) => 0; + + // one last check after power-cycle + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "dummy") => 0; + lfs_stat(&lfs, "dummy", &info) => 0; + assert(strcmp(info.name, "dummy") == 0); + lfs_unmount(&lfs) => 0; +''' +define.BLOCK_CYCLES = [32, 33, 1] +define.N = [10, 100, 1000] + +[[case]] # expanding superblock with power cycle +code = ''' + lfs_format(&lfs, &cfg) => 0; + for (int i = 0; i < N; i++) { + lfs_mount(&lfs, &cfg) => 0; + // remove lingering dummy? + err = lfs_remove(&lfs, "dummy"); + assert(err == 0 || (err == LFS_ERR_NOENT && i == 0)); + + lfs_mkdir(&lfs, "dummy") => 0; + lfs_stat(&lfs, "dummy", &info) => 0; + assert(strcmp(info.name, "dummy") == 0); + lfs_unmount(&lfs) => 0; + } + + // one last check after power-cycle + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "dummy", &info) => 0; + assert(strcmp(info.name, "dummy") == 0); + lfs_unmount(&lfs) => 0; +''' +define.BLOCK_CYCLES = [32, 33, 1] +define.N = [10, 100, 1000] + +[[case]] # reentrant expanding superblock +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + for (int i = 0; i < N; i++) { + // remove lingering dummy? + err = lfs_remove(&lfs, "dummy"); + assert(err == 0 || (err == LFS_ERR_NOENT && i == 0)); + + lfs_mkdir(&lfs, "dummy") => 0; + lfs_stat(&lfs, "dummy", &info) => 0; + assert(strcmp(info.name, "dummy") == 0); + } + + lfs_unmount(&lfs) => 0; + + // one last check after power-cycle + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "dummy", &info) => 0; + assert(strcmp(info.name, "dummy") == 0); + lfs_unmount(&lfs) => 0; +''' +define.BLOCK_CYCLES = [2, 1] +define.N = 24 +reentrant = true From b06ce542791d0d3d0ff56fdb762404e9ce5bedea Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 12 Jan 2020 22:21:09 -0600 Subject: [PATCH 06/41] Migrated the bulk of the feature-specific tests This involved some minor tweaks for the various types of tests, added predicates to the test framework (necessary for test_entries and test_alloc), and cleaned up some of the testing semantics such as reporting how many tests are filtered, showing permutation config on the result screen, and properly inheriting suite config in cases. --- .gitignore | 1 + scripts/test_.py | 126 ++++--- tests_/test_alloc.toml | 566 +++++++++++++++++++++++++++++++ tests_/test_attrs.toml | 305 +++++++++++++++++ tests_/test_dirs.toml | 122 +++++++ tests_/test_entries.toml | 611 ++++++++++++++++++++++++++++++++++ tests_/test_interspersed.toml | 262 +++++++++++++++ tests_/test_paths.toml | 294 ++++++++++++++++ tests_/test_seek.toml | 380 +++++++++++++++++++++ tests_/test_truncate.toml | 395 ++++++++++++++++++++++ 10 files changed, 3024 insertions(+), 38 deletions(-) create mode 100644 tests_/test_alloc.toml create mode 100644 tests_/test_attrs.toml create mode 100644 tests_/test_entries.toml create mode 100644 tests_/test_interspersed.toml create mode 100644 tests_/test_paths.toml create mode 100644 tests_/test_seek.toml create mode 100644 tests_/test_truncate.toml diff --git a/.gitignore b/.gitignore index 36f92cd8..1e12cca5 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,4 @@ blocks/ lfs test.c +tests_/*.toml.* diff --git a/scripts/test_.py b/scripts/test_.py index 2f016e43..ab674c97 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -4,6 +4,20 @@ # .toml files stored in the tests directory. # +# TODO +# - nargs > 1? +# x show perm config on failure +# x filtering +# n show perm config on verbose? +# - better lineno tracking for cases? +# n non-int perms? +# - different path format? +# - suite.prologue, suite.epilogue +# - in +# x change BLOCK_CYCLES to -1 by default +# x change persist behaviour +# x config chaining correct + import toml import glob import re @@ -20,7 +34,7 @@ TESTDIR = 'tests_' RULES = """ define FLATTEN -%$(subst /,.,$(target:.c=.t.c)): $(target) +%$(subst /,.,$(target:.c=.tc)): $(target) cat <(echo '#line 1 "$$<"') $$< > $$@ endef $(foreach target,$(SRC),$(eval $(FLATTEN))) @@ -28,12 +42,12 @@ -include tests_/*.d .SECONDARY: -%.c: %.t.c +%.c: %.tc ./scripts/explode_asserts.py $< -o $@ %.test: override CFLAGS += -fdiagnostics-color=always %.test: override CFLAGS += -ggdb -%.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.test.$f) +%.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.$f) $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ """ GLOBALS = """ @@ -49,7 +63,7 @@ "LFS_PROG_SIZE": "LFS_READ_SIZE", "LFS_BLOCK_SIZE": 512, "LFS_BLOCK_COUNT": 1024, - "LFS_BLOCK_CYCLES": 1024, + "LFS_BLOCK_CYCLES": -1, "LFS_CACHE_SIZE": "(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)", "LFS_LOOKAHEAD_SIZE": 16, "LFS_ERASE_VALUE": 0xff, @@ -122,12 +136,19 @@ def __init__(self, config, suite=None, caseno=None, lineno=None, **_): self.code = config['code'] self.defines = config.get('define', {}) + self.if_ = config.get('if', None) self.leaky = config.get('leaky', False) def __str__(self): if hasattr(self, 'permno'): - return '%s[%d,%d]' % ( - self.suite.name, self.caseno, self.permno) + if any(k not in self.case.defines for k in self.defines): + return '%s[%d,%d] (%s)' % ( + self.suite.name, self.caseno, self.permno, ', '.join( + '%s=%s' % (k, v) for k, v in self.defines.items() + if k not in self.case.defines)) + else: + return '%s[%d,%d]' % ( + self.suite.name, self.caseno, self.permno) else: return '%s[%d]' % ( self.suite.name, self.caseno) @@ -176,12 +197,26 @@ def build(self, f, **_): f.write('}\n') + def shouldtest(self, **args): + if self.if_ is not None: + return eval(self.if_, None, self.defines.copy()) + else: + return True + def test(self, exec=[], persist=False, gdb=False, failure=None, **args): # build command cmd = exec + ['./%s.test' % self.suite.path, repr(self.caseno), repr(self.permno)] + + # persist disk or keep in RAM for speed? if persist: - cmd.append(self.suite.path + '.test.disk') + if persist != 'noerase': + try: + os.remove(self.suite.path + '.disk') + except FileNotFoundError: + pass + + cmd.append(self.suite.path + '.disk') # failed? drop into debugger? if gdb and failure: @@ -244,10 +279,10 @@ def __init__(self, config, **args): self.leaky = config.get('leaky', False) super().__init__(config, **args) - def test(self, exec=[], **args): - if self.leaky: - return + def shouldtest(self, **args): + return not self.leaky and super().shouldtest(**args) + def test(self, exec=[], **args): exec = exec + [ 'valgrind', '--leak-check=full', @@ -260,14 +295,14 @@ def __init__(self, config, **args): self.reentrant = config.get('reentrant', False) super().__init__(config, **args) - def test(self, exec=[], persist=False, gdb=False, failure=None, **args): - if not self.reentrant: - return + def shouldtest(self, **args): + return self.reentrant and super().shouldtest(**args) + def test(self, exec=[], persist=False, gdb=False, failure=None, **args): # clear disk first? - if not persist: + if persist != 'noerase': try: - os.remove(self.suite.path + '.test.disk') + os.remove(self.suite.path + '.disk') except FileNotFoundError: pass @@ -293,7 +328,7 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): '33', '--args'] try: - return super().test(exec=nexec, persist=True, **args) + return super().test(exec=nexec, persist='noerase', **args) except TestFailure as nfailure: if nfailure.returncode == 33: continue @@ -326,6 +361,11 @@ def __init__(self, path, TestCase=TestCase, **args): # create initial test cases self.cases = [] for i, (case, lineno) in enumerate(zip(config['case'], linenos)): + # give our case's config a copy of our "global" config + for k, v in config.items(): + if k not in case: + case[k] = v + # initialize test case self.cases.append(self.TestCase(case, suite=self, caseno=i, lineno=lineno, **args)) @@ -430,7 +470,7 @@ def build(self, **args): # add test-related rules rules = RULES.replace(4*' ', '\t') - with open(self.path + '.test.mk', 'w') as mk: + with open(self.path + '.mk', 'w') as mk: mk.write(rules) mk.write('\n') @@ -440,13 +480,13 @@ def build(self, **args): self.path+'.test', k, v)) # write test.c in base64 so make can decide when to rebuild - mk.write('%s: %s\n' % (self.path+'.test.t.c', self.path)) + mk.write('%s: %s\n' % (self.path+'.test.tc', self.path)) mk.write('\t@base64 -d <<< ') mk.write(base64.b64encode( f.getvalue().encode('utf8')).decode('utf8')) mk.write(' > $@\n') - self.makefile = self.path + '.test.mk' + self.makefile = self.path + '.mk' self.target = self.path + '.test' return self.makefile, self.target @@ -460,6 +500,8 @@ def test(self, caseno=None, permno=None, **args): continue if permno is not None and perm.permno != permno: continue + if not perm.shouldtest(**args): + continue try: result = perm.test(**args) @@ -473,11 +515,10 @@ def test(self, caseno=None, permno=None, **args): sys.stdout.write('\n') raise else: - if result == PASS: - perm.result = PASS - if not args.get('verbose', True): - sys.stdout.write(PASS) - sys.stdout.flush() + perm.result = PASS + if not args.get('verbose', True): + sys.stdout.write(PASS) + sys.stdout.flush() if not args.get('verbose', True): sys.stdout.write('\n') @@ -581,6 +622,13 @@ def main(**args): sum(len(suite.cases) for suite in suites), sum(len(suite.perms) for suite in suites))) + filtered = 0 + for suite in suites: + for perm in suite.perms: + filtered += perm.shouldtest(**args) + if filtered != sum(len(suite.perms) for suite in suites): + print('filtered down to %d permutations' % filtered) + print('====== testing ======') try: for suite in suites: @@ -588,18 +636,6 @@ def main(**args): except TestFailure: pass - if args.get('gdb', False): - failure = None - for suite in suites: - for perm in suite.perms: - if getattr(perm, 'result', PASS) != PASS: - failure = perm.result - if failure is not None: - print('======= gdb ======') - # drop into gdb - failure.case.test(failure=failure, **args) - sys.exit(0) - print('====== results ======') passed = 0 failed = 0 @@ -633,6 +669,19 @@ def main(**args): sys.stdout.write('\n') failed += 1 + if args.get('gdb', False): + failure = None + for suite in suites: + for perm in suite.perms: + if getattr(perm, 'result', PASS) != PASS: + failure = perm.result + if failure is not None: + print('======= gdb ======') + # drop into gdb + failure.case.test(failure=failure, **args) + sys.exit(0) + + print('tests passed: %d' % passed) print('tests failed: %d' % failed) @@ -652,8 +701,9 @@ def main(**args): help="Output everything that is happening.") parser.add_argument('-k', '--keep-going', action='store_true', help="Run all tests instead of stopping on first error. Useful for CI.") - parser.add_argument('-p', '--persist', action='store_true', - help="Don't reset the tests disk before each test.") + parser.add_argument('-p', '--persist', choices=['erase', 'noerase'], + nargs='?', const='erase', + help="Store disk image in a file.") parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], nargs='?', const='assert', help="Drop into gdb on test failure.") diff --git a/tests_/test_alloc.toml b/tests_/test_alloc.toml new file mode 100644 index 00000000..6851e772 --- /dev/null +++ b/tests_/test_alloc.toml @@ -0,0 +1,566 @@ +# allocator tests +# note for these to work there are many constraints on the device geometry + +[[case]] # parallel allocation test +code = ''' + const char *names[FILES] = {"bacon", "eggs", "pancakes"}; + lfs_file_t files[FILES]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "breakfast") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &files[n], path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + } + for (int n = 0; n < FILES; n++) { + size = strlen(names[n]); + for (lfs_size_t i = 0; i < SIZE; i += size) { + lfs_file_write(&lfs, &files[n], names[n], size) => size; + } + } + for (int n = 0; n < FILES; n++) { + lfs_file_close(&lfs, &files[n]) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + size = strlen(names[n]); + for (lfs_size_t i = 0; i < SIZE; i += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, names[n], size) == 0); + } + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; +''' +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' + +[[case]] # serial allocation test +code = ''' + const char *names[FILES] = {"bacon", "eggs", "pancakes"}; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "breakfast") => 0; + lfs_unmount(&lfs) => 0; + + for (int n = 0; n < FILES; n++) { + lfs_mount(&lfs, &cfg) => 0; + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + size = strlen(names[n]); + memcpy(buffer, names[n], size); + for (int i = 0; i < SIZE; i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + } + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + size = strlen(names[n]); + for (int i = 0; i < SIZE; i += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, names[n], size) == 0); + } + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; +''' +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' + +[[case]] # parallel allocation reuse test +code = ''' + const char *names[FILES] = {"bacon", "eggs", "pancakes"}; + lfs_file_t files[FILES]; + + lfs_format(&lfs, &cfg) => 0; + + for (int c = 0; c < CYCLES; c++) { + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "breakfast") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &files[n], path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + } + for (int n = 0; n < FILES; n++) { + size = strlen(names[n]); + for (int i = 0; i < SIZE; i += size) { + lfs_file_write(&lfs, &files[n], names[n], size) => size; + } + } + for (int n = 0; n < FILES; n++) { + lfs_file_close(&lfs, &files[n]) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + size = strlen(names[n]); + for (int i = 0; i < SIZE; i += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, names[n], size) == 0); + } + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_remove(&lfs, path) => 0; + } + lfs_remove(&lfs, "breakfast") => 0; + lfs_unmount(&lfs) => 0; + } +''' +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.CYCLES = [1, 10] + +[[case]] # serial allocation reuse test +code = ''' + const char *names[FILES] = {"bacon", "eggs", "pancakes"}; + + lfs_format(&lfs, &cfg) => 0; + + for (int c = 0; c < CYCLES; c++) { + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "breakfast") => 0; + lfs_unmount(&lfs) => 0; + + for (int n = 0; n < FILES; n++) { + lfs_mount(&lfs, &cfg) => 0; + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + size = strlen(names[n]); + memcpy(buffer, names[n], size); + for (int i = 0; i < SIZE; i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + } + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + size = strlen(names[n]); + for (int i = 0; i < SIZE; i += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, names[n], size) == 0); + } + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int n = 0; n < FILES; n++) { + sprintf(path, "breakfast/%s", names[n]); + lfs_remove(&lfs, path) => 0; + } + lfs_remove(&lfs, "breakfast") => 0; + lfs_unmount(&lfs) => 0; + } +''' +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.CYCLES = [1, 10] + +[[case]] # exhaustion test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + size = strlen("exhaustion"); + memcpy(buffer, "exhaustion", size); + lfs_file_write(&lfs, &file, buffer, size) => size; + lfs_file_sync(&lfs, &file) => 0; + + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + lfs_ssize_t res; + while (true) { + res = lfs_file_write(&lfs, &file, buffer, size); + if (res < 0) { + break; + } + + res => size; + } + res => LFS_ERR_NOSPC; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY); + size = strlen("exhaustion"); + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "exhaustion", size) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # exhaustion wraparound test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + lfs_file_open(&lfs, &file, "padding", LFS_O_WRONLY | LFS_O_CREAT); + size = strlen("buffering"); + memcpy(buffer, "buffering", size); + for (int i = 0; i < SIZE; i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + lfs_remove(&lfs, "padding") => 0; + + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + size = strlen("exhaustion"); + memcpy(buffer, "exhaustion", size); + lfs_file_write(&lfs, &file, buffer, size) => size; + lfs_file_sync(&lfs, &file) => 0; + + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + lfs_ssize_t res; + while (true) { + res = lfs_file_write(&lfs, &file, buffer, size); + if (res < 0) { + break; + } + + res => size; + } + res => LFS_ERR_NOSPC; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY); + size = strlen("exhaustion"); + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "exhaustion", size) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_remove(&lfs, "exhaustion") => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / 3)' + +[[case]] # dir exhaustion test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // find out max file size + lfs_mkdir(&lfs, "exhaustiondir") => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + int count = 0; + while (true) { + err = lfs_file_write(&lfs, &file, buffer, size); + if (err < 0) { + break; + } + + count += 1; + } + err => LFS_ERR_NOSPC; + lfs_file_close(&lfs, &file) => 0; + + lfs_remove(&lfs, "exhaustion") => 0; + lfs_remove(&lfs, "exhaustiondir") => 0; + + // see if dir fits with max file size + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + for (int i = 0; i < count; i++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + lfs_mkdir(&lfs, "exhaustiondir") => 0; + lfs_remove(&lfs, "exhaustiondir") => 0; + lfs_remove(&lfs, "exhaustion") => 0; + + // see if dir fits with > max file size + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + for (int i = 0; i < count+1; i++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + lfs_mkdir(&lfs, "exhaustiondir") => LFS_ERR_NOSPC; + + lfs_remove(&lfs, "exhaustion") => 0; + lfs_unmount(&lfs) => 0; +''' + +# Below, I don't like these tests. They're fragile and depend _heavily_ +# on the geometry of the block device. But they are valuable. Eventually they +# should be removed and replaced with generalized tests. + +[[case]] # chained dir exhaustion test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // find out max file size + lfs_mkdir(&lfs, "exhaustiondir") => 0; + for (int i = 0; i < 10; i++) { + sprintf(path, "dirwithanexhaustivelylongnameforpadding%d", i); + lfs_mkdir(&lfs, path) => 0; + } + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + int count = 0; + while (true) { + err = lfs_file_write(&lfs, &file, buffer, size); + if (err < 0) { + break; + } + + count += 1; + } + err => LFS_ERR_NOSPC; + lfs_file_close(&lfs, &file) => 0; + + lfs_remove(&lfs, "exhaustion") => 0; + lfs_remove(&lfs, "exhaustiondir") => 0; + for (int i = 0; i < 10; i++) { + sprintf(path, "dirwithanexhaustivelylongnameforpadding%d", i); + lfs_remove(&lfs, path) => 0; + } + + // see that chained dir fails + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + for (int i = 0; i < count+1; i++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_sync(&lfs, &file) => 0; + + for (int i = 0; i < 10; i++) { + sprintf(path, "dirwithanexhaustivelylongnameforpadding%d", i); + lfs_mkdir(&lfs, path) => 0; + } + + lfs_mkdir(&lfs, "exhaustiondir") => LFS_ERR_NOSPC; + + // shorten file to try a second chained dir + while (true) { + err = lfs_mkdir(&lfs, "exhaustiondir"); + if (err != LFS_ERR_NOSPC) { + break; + } + + lfs_ssize_t filesize = lfs_file_size(&lfs, &file); + filesize > 0 => true; + + lfs_file_truncate(&lfs, &file, filesize - size) => 0; + lfs_file_sync(&lfs, &file) => 0; + } + err => 0; + + lfs_mkdir(&lfs, "exhaustiondir2") => LFS_ERR_NOSPC; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.LFS_BLOCK_SIZE = 512 +define.LFS_BLOCK_COUNT = 1024 +if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' + +[[case]] # split dir test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // create one block hole for half a directory + lfs_file_open(&lfs, &file, "bump", LFS_O_WRONLY | LFS_O_CREAT) => 0; + for (lfs_size_t i = 0; i < cfg.block_size; i += 2) { + memcpy(&buffer[i], "hi", 2); + } + lfs_file_write(&lfs, &file, buffer, cfg.block_size) => cfg.block_size; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < (cfg.block_count-4)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + // remount to force reset of lookahead + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // open hole + lfs_remove(&lfs, "bump") => 0; + + lfs_mkdir(&lfs, "splitdir") => 0; + lfs_file_open(&lfs, &file, "splitdir/bump", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + for (lfs_size_t i = 0; i < cfg.block_size; i += 2) { + memcpy(&buffer[i], "hi", 2); + } + lfs_file_write(&lfs, &file, buffer, 2*cfg.block_size) => LFS_ERR_NOSPC; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' +define.LFS_BLOCK_SIZE = 512 +define.LFS_BLOCK_COUNT = 1024 +if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' + +[[case]] # outdated lookahead test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // fill completely with two files + lfs_file_open(&lfs, &file, "exhaustion1", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2)/2)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "exhaustion2", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2+1)/2)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + // remount to force reset of lookahead + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // rewrite one file + lfs_file_open(&lfs, &file, "exhaustion1", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_sync(&lfs, &file) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2)/2)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + // rewrite second file, this requires lookahead does not + // use old population + lfs_file_open(&lfs, &file, "exhaustion2", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_sync(&lfs, &file) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2+1)/2)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; +''' +define.LFS_BLOCK_SIZE = 512 +define.LFS_BLOCK_COUNT = 1024 +if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' + +[[case]] # outdated lookahead and split dir test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // fill completely with two files + lfs_file_open(&lfs, &file, "exhaustion1", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2)/2)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "exhaustion2", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2+1)/2)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + // remount to force reset of lookahead + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // rewrite one file with a hole of one block + lfs_file_open(&lfs, &file, "exhaustion1", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_sync(&lfs, &file) => 0; + size = strlen("blahblahblahblah"); + memcpy(buffer, "blahblahblahblah", size); + for (lfs_size_t i = 0; + i < ((cfg.block_count-2)/2 - 1)*(cfg.block_size-8); + i += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + + // try to allocate a directory, should fail! + lfs_mkdir(&lfs, "split") => LFS_ERR_NOSPC; + + // file should not fail + lfs_file_open(&lfs, &file, "notasplit", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "hi", 2) => 2; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' +define.LFS_BLOCK_SIZE = 512 +define.LFS_BLOCK_COUNT = 1024 +if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' diff --git a/tests_/test_attrs.toml b/tests_/test_attrs.toml new file mode 100644 index 00000000..f1254183 --- /dev/null +++ b/tests_/test_attrs.toml @@ -0,0 +1,305 @@ +[[case]] # set/get attribute +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "hello") => 0; + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "hello", strlen("hello")) => strlen("hello"); + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(buffer, 0, sizeof(buffer)); + lfs_setattr(&lfs, "hello", 'A', "aaaa", 4) => 0; + lfs_setattr(&lfs, "hello", 'B', "bbbbbb", 6) => 0; + lfs_setattr(&lfs, "hello", 'C', "ccccc", 5) => 0; + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 6; + lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "bbbbbb", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "hello", 'B', "", 0) => 0; + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 0; + lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_removeattr(&lfs, "hello", 'B') => 0; + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => LFS_ERR_NOATTR; + lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "hello", 'B', "dddddd", 6) => 0; + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 6; + lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "dddddd", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "hello", 'B', "eee", 3) => 0; + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 3; + lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "eee\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "hello", 'A', buffer, LFS_ATTR_MAX+1) => LFS_ERR_NOSPC; + lfs_setattr(&lfs, "hello", 'B', "fffffffff", 9) => 0; + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 9; + lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; + + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(buffer, 0, sizeof(buffer)); + lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "hello", 'B', buffer+4, 9) => 9; + lfs_getattr(&lfs, "hello", 'C', buffer+13, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "fffffffff", 9) => 0; + memcmp(buffer+13, "ccccc", 5) => 0; + + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => strlen("hello"); + memcmp(buffer, "hello", strlen("hello")) => 0; + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # set/get root attribute +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "hello") => 0; + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "hello", strlen("hello")) => strlen("hello"); + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + + memset(buffer, 0, sizeof(buffer)); + lfs_setattr(&lfs, "/", 'A', "aaaa", 4) => 0; + lfs_setattr(&lfs, "/", 'B', "bbbbbb", 6) => 0; + lfs_setattr(&lfs, "/", 'C', "ccccc", 5) => 0; + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 6; + lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "bbbbbb", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "/", 'B', "", 0) => 0; + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 0; + lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_removeattr(&lfs, "/", 'B') => 0; + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => LFS_ERR_NOATTR; + lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "/", 'B', "dddddd", 6) => 0; + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 6; + lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "dddddd", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "/", 'B', "eee", 3) => 0; + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 3; + lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "eee\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + lfs_setattr(&lfs, "/", 'A', buffer, LFS_ATTR_MAX+1) => LFS_ERR_NOSPC; + lfs_setattr(&lfs, "/", 'B', "fffffffff", 9) => 0; + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 9; + lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(buffer, 0, sizeof(buffer)); + lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; + lfs_getattr(&lfs, "/", 'B', buffer+4, 9) => 9; + lfs_getattr(&lfs, "/", 'C', buffer+13, 5) => 5; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "fffffffff", 9) => 0; + memcmp(buffer+13, "ccccc", 5) => 0; + + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => strlen("hello"); + memcmp(buffer, "hello", strlen("hello")) => 0; + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # set/get file attribute +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "hello") => 0; + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "hello", strlen("hello")) => strlen("hello"); + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(buffer, 0, sizeof(buffer)); + struct lfs_attr attrs1[] = { + {'A', buffer, 4}, + {'B', buffer+4, 6}, + {'C', buffer+10, 5}, + }; + struct lfs_file_config cfg1 = {.attrs=attrs1, .attr_count=3}; + + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; + memcpy(buffer, "aaaa", 4); + memcpy(buffer+4, "bbbbbb", 6); + memcpy(buffer+10, "ccccc", 5); + lfs_file_close(&lfs, &file) => 0; + memset(buffer, 0, 15); + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; + lfs_file_close(&lfs, &file) => 0; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "bbbbbb", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + attrs1[1].size = 0; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; + lfs_file_close(&lfs, &file) => 0; + memset(buffer, 0, 15); + attrs1[1].size = 6; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; + lfs_file_close(&lfs, &file) => 0; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + attrs1[1].size = 6; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; + memcpy(buffer+4, "dddddd", 6); + lfs_file_close(&lfs, &file) => 0; + memset(buffer, 0, 15); + attrs1[1].size = 6; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; + lfs_file_close(&lfs, &file) => 0; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "dddddd", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + attrs1[1].size = 3; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; + memcpy(buffer+4, "eee", 3); + lfs_file_close(&lfs, &file) => 0; + memset(buffer, 0, 15); + attrs1[1].size = 6; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; + lfs_file_close(&lfs, &file) => 0; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "eee\0\0\0", 6) => 0; + memcmp(buffer+10, "ccccc", 5) => 0; + + attrs1[0].size = LFS_ATTR_MAX+1; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) + => LFS_ERR_NOSPC; + + struct lfs_attr attrs2[] = { + {'A', buffer, 4}, + {'B', buffer+4, 9}, + {'C', buffer+13, 5}, + }; + struct lfs_file_config cfg2 = {.attrs=attrs2, .attr_count=3}; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDWR, &cfg2) => 0; + memcpy(buffer+4, "fffffffff", 9); + lfs_file_close(&lfs, &file) => 0; + attrs1[0].size = 4; + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(buffer, 0, sizeof(buffer)); + struct lfs_attr attrs3[] = { + {'A', buffer, 4}, + {'B', buffer+4, 9}, + {'C', buffer+13, 5}, + }; + struct lfs_file_config cfg3 = {.attrs=attrs3, .attr_count=3}; + + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg3) => 0; + lfs_file_close(&lfs, &file) => 0; + memcmp(buffer, "aaaa", 4) => 0; + memcmp(buffer+4, "fffffffff", 9) => 0; + memcmp(buffer+13, "ccccc", 5) => 0; + + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => strlen("hello"); + memcmp(buffer, "hello", strlen("hello")) => 0; + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # deferred file attributes +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "hello") => 0; + lfs_file_open(&lfs, &file, "hello/hello", LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "hello", strlen("hello")) => strlen("hello"); + lfs_file_close(&lfs, &file); + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_setattr(&lfs, "hello/hello", 'B', "fffffffff", 9) => 0; + lfs_setattr(&lfs, "hello/hello", 'C', "ccccc", 5) => 0; + + memset(buffer, 0, sizeof(buffer)); + struct lfs_attr attrs1[] = { + {'B', "gggg", 4}, + {'C', "", 0}, + {'D', "hhhh", 4}, + }; + struct lfs_file_config cfg1 = {.attrs=attrs1, .attr_count=3}; + + lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; + + lfs_getattr(&lfs, "hello/hello", 'B', buffer, 9) => 9; + lfs_getattr(&lfs, "hello/hello", 'C', buffer+9, 9) => 5; + lfs_getattr(&lfs, "hello/hello", 'D', buffer+18, 9) => LFS_ERR_NOATTR; + memcmp(buffer, "fffffffff", 9) => 0; + memcmp(buffer+9, "ccccc\0\0\0\0", 9) => 0; + memcmp(buffer+18, "\0\0\0\0\0\0\0\0\0", 9) => 0; + + lfs_file_sync(&lfs, &file) => 0; + lfs_getattr(&lfs, "hello/hello", 'B', buffer, 9) => 4; + lfs_getattr(&lfs, "hello/hello", 'C', buffer+9, 9) => 0; + lfs_getattr(&lfs, "hello/hello", 'D', buffer+18, 9) => 4; + memcmp(buffer, "gggg\0\0\0\0\0", 9) => 0; + memcmp(buffer+9, "\0\0\0\0\0\0\0\0\0", 9) => 0; + memcmp(buffer+18, "hhhh\0\0\0\0\0", 9) => 0; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' diff --git a/tests_/test_dirs.toml b/tests_/test_dirs.toml index e3e04533..65987538 100644 --- a/tests_/test_dirs.toml +++ b/tests_/test_dirs.toml @@ -714,3 +714,125 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; ''' + +[[case]] # directory seek +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "hello") => 0; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "hello/kitty%03d", i); + lfs_mkdir(&lfs, path) => 0; + } + lfs_unmount(&lfs) => 0; + + for (int j = 2; j < COUNT; j++) { + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "hello") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_soff_t pos; + for (int i = 0; i < j; i++) { + sprintf(path, "kitty%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + pos = lfs_dir_tell(&lfs, &dir); + assert(pos >= 0); + } + + lfs_dir_seek(&lfs, &dir, pos) => 0; + sprintf(path, "kitty%03d", j); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_rewind(&lfs, &dir) => 0; + sprintf(path, "kitty%03d", 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_seek(&lfs, &dir, pos) => 0; + sprintf(path, "kitty%03d", j); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + } +''' +define.COUNT = [4, 128, 132] + +[[case]] # root seek +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "hi%03d", i); + lfs_mkdir(&lfs, path) => 0; + } + lfs_unmount(&lfs) => 0; + + for (int j = 2; j < COUNT; j++) { + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_soff_t pos; + for (int i = 0; i < j; i++) { + sprintf(path, "hi%03d", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + pos = lfs_dir_tell(&lfs, &dir); + assert(pos >= 0); + } + + lfs_dir_seek(&lfs, &dir, pos) => 0; + sprintf(path, "hi%03d", j); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_rewind(&lfs, &dir) => 0; + sprintf(path, "hi%03d", 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_seek(&lfs, &dir, pos) => 0; + sprintf(path, "hi%03d", j); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + } +''' +define.COUNT = [4, 128, 132] + diff --git a/tests_/test_entries.toml b/tests_/test_entries.toml new file mode 100644 index 00000000..cbbf6a1d --- /dev/null +++ b/tests_/test_entries.toml @@ -0,0 +1,611 @@ +# These tests are for some specific corner cases with neighboring inline files. +# Note that these tests are intended for 512 byte inline sizes. They should +# still pass with other inline sizes but wouldn't be testing anything. + +define.LFS_CACHE_SIZE = 512 +if = 'LFS_CACHE_SIZE == 512' + +[[case]] # entry grow test +code = ''' + uint8_t wbuffer[1024]; + uint8_t rbuffer[1024]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // write hi0 20 + sprintf(path, "hi0"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi2 20 + sprintf(path, "hi2"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi3 20 + sprintf(path, "hi3"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // write hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi0 20 + sprintf(path, "hi0"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi2 20 + sprintf(path, "hi2"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 20 + sprintf(path, "hi3"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' + +[[case]] # entry shrink test +code = ''' + uint8_t wbuffer[1024]; + uint8_t rbuffer[1024]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // write hi0 20 + sprintf(path, "hi0"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi2 20 + sprintf(path, "hi2"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi3 20 + sprintf(path, "hi3"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // write hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi0 20 + sprintf(path, "hi0"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi2 20 + sprintf(path, "hi2"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 20 + sprintf(path, "hi3"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' + +[[case]] # entry spill test +code = ''' + uint8_t wbuffer[1024]; + uint8_t rbuffer[1024]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // write hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' + +[[case]] # entry push spill test +code = ''' + uint8_t wbuffer[1024]; + uint8_t rbuffer[1024]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // write hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // write hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' + +[[case]] # entry push spill two test +code = ''' + uint8_t wbuffer[1024]; + uint8_t rbuffer[1024]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // write hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi4 200 + sprintf(path, "hi4"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi1 20 + sprintf(path, "hi1"); size = 20; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // write hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + // read hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi4 200 + sprintf(path, "hi4"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' + +[[case]] # entry drop test +code = ''' + uint8_t wbuffer[1024]; + uint8_t rbuffer[1024]; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // write hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi1 200 + sprintf(path, "hi1"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + // write hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_close(&lfs, &file) => 0; + + lfs_remove(&lfs, "hi1") => 0; + lfs_stat(&lfs, "hi1", &info) => LFS_ERR_NOENT; + // read hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi2 200 + sprintf(path, "hi2"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_remove(&lfs, "hi2") => 0; + lfs_stat(&lfs, "hi2", &info) => LFS_ERR_NOENT; + // read hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + // read hi3 200 + sprintf(path, "hi3"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_remove(&lfs, "hi3") => 0; + lfs_stat(&lfs, "hi3", &info) => LFS_ERR_NOENT; + // read hi0 200 + sprintf(path, "hi0"); size = 200; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => size; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_remove(&lfs, "hi0") => 0; + lfs_stat(&lfs, "hi0", &info) => LFS_ERR_NOENT; + + lfs_unmount(&lfs) => 0; +''' + +[[case]] # create too big +code = ''' + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(path, 'm', 200); + path[200] = '\0'; + + size = 400; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + uint8_t wbuffer[1024]; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_close(&lfs, &file) => 0; + + size = 400; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # resize too big +code = ''' + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(path, 'm', 200); + path[200] = '\0'; + + size = 40; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + uint8_t wbuffer[1024]; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_close(&lfs, &file) => 0; + + size = 40; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + + size = 400; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + memset(wbuffer, 'c', size); + lfs_file_write(&lfs, &file, wbuffer, size) => size; + lfs_file_close(&lfs, &file) => 0; + + size = 400; + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(rbuffer, wbuffer, size) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' diff --git a/tests_/test_interspersed.toml b/tests_/test_interspersed.toml new file mode 100644 index 00000000..ef4f9d24 --- /dev/null +++ b/tests_/test_interspersed.toml @@ -0,0 +1,262 @@ + +[[case]] # interspersed file test +code = ''' + lfs_file_t files[FILES]; + const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_file_open(&lfs, &files[j], path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + } + + for (int i = 0; i < SIZE; i++) { + for (int j = 0; j < FILES; j++) { + lfs_file_write(&lfs, &files[j], &alphas[j], 1) => 1; + } + } + + for (int j = 0; j < FILES; j++) { + lfs_file_close(&lfs, &files[j]); + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_file_open(&lfs, &files[j], path, LFS_O_RDONLY) => 0; + } + + for (int i = 0; i < 10; i++) { + for (int j = 0; j < FILES; j++) { + lfs_file_read(&lfs, &files[j], buffer, 1) => 1; + assert(buffer[0] == alphas[j]); + } + } + + for (int j = 0; j < FILES; j++) { + lfs_file_close(&lfs, &files[j]); + } + + lfs_unmount(&lfs) => 0; +''' +# TODO FILES=26 found bug +#define.SIZE = [10, 100] +#define.FILES = [4, 10, 26] +define = [ + {SIZE=10, FILES=4}, + {SIZE=10, FILES=10}, + #{SIZE=10, FILES=26}, + {SIZE=100, FILES=4}, + {SIZE=100, FILES=10}, + #{SIZE=100, FILES=26}, +] + +[[case]] # interspersed remove file test +code = ''' + const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + for (int i = 0; i < SIZE; i++) { + lfs_file_write(&lfs, &file, &alphas[j], 1) => 1; + } + lfs_file_close(&lfs, &file); + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "zzz", LFS_O_WRONLY | LFS_O_CREAT) => 0; + for (int j = 0; j < FILES; j++) { + lfs_file_write(&lfs, &file, (const void*)"~", 1) => 1; + lfs_file_sync(&lfs, &file) => 0; + + sprintf(path, "%c", alphas[j]); + lfs_remove(&lfs, path) => 0; + } + lfs_file_close(&lfs, &file); + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "zzz") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == FILES); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "zzz", LFS_O_RDONLY) => 0; + for (int i = 0; i < FILES; i++) { + lfs_file_read(&lfs, &file, buffer, 1) => 1; + assert(buffer[0] == '~'); + } + lfs_file_close(&lfs, &file); + + lfs_unmount(&lfs) => 0; +''' +define.SIZE = [10, 100] +define.FILES = [4, 10, 26] + +[[case]] # remove inconveniently test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_t files[3]; + lfs_file_open(&lfs, &files[0], "e", LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_open(&lfs, &files[1], "f", LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_open(&lfs, &files[2], "g", LFS_O_WRONLY | LFS_O_CREAT) => 0; + + for (int i = 0; i < SIZE/2; i++) { + lfs_file_write(&lfs, &files[0], (const void*)"e", 1) => 1; + lfs_file_write(&lfs, &files[1], (const void*)"f", 1) => 1; + lfs_file_write(&lfs, &files[2], (const void*)"g", 1) => 1; + } + + lfs_remove(&lfs, "f") => 0; + + for (int i = 0; i < SIZE/2; i++) { + lfs_file_write(&lfs, &files[0], (const void*)"e", 1) => 1; + lfs_file_write(&lfs, &files[1], (const void*)"f", 1) => 1; + lfs_file_write(&lfs, &files[2], (const void*)"g", 1) => 1; + } + + lfs_file_close(&lfs, &files[0]); + lfs_file_close(&lfs, &files[1]); + lfs_file_close(&lfs, &files[2]); + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "e") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "g") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &files[0], "e", LFS_O_RDONLY) => 0; + lfs_file_open(&lfs, &files[1], "g", LFS_O_RDONLY) => 0; + for (int i = 0; i < SIZE; i++) { + lfs_file_read(&lfs, &files[0], buffer, 1) => 1; + assert(buffer[0] == 'e'); + lfs_file_read(&lfs, &files[1], buffer, 1) => 1; + assert(buffer[0] == 'g'); + } + lfs_file_close(&lfs, &files[0]); + lfs_file_close(&lfs, &files[1]); + + lfs_unmount(&lfs) => 0; +''' +define.SIZE = [10, 100] + +[[case]] # reentrant interspersed file test +code = ''' + lfs_file_t files[FILES]; + const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; + + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_file_open(&lfs, &files[j], path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + } + + for (int i = 0; i < SIZE; i++) { + for (int j = 0; j < FILES; j++) { + size = lfs_file_size(&lfs, &files[j]); + assert((int)size >= 0); + if ((int)size <= i) { + lfs_file_write(&lfs, &files[j], &alphas[j], 1) => 1; + lfs_file_sync(&lfs, &files[j]) => 0; + } + } + } + + for (int j = 0; j < FILES; j++) { + lfs_file_close(&lfs, &files[j]); + } + + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, path) == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int j = 0; j < FILES; j++) { + sprintf(path, "%c", alphas[j]); + lfs_file_open(&lfs, &files[j], path, LFS_O_RDONLY) => 0; + } + + for (int i = 0; i < 10; i++) { + for (int j = 0; j < FILES; j++) { + lfs_file_read(&lfs, &files[j], buffer, 1) => 1; + assert(buffer[0] == alphas[j]); + } + } + + for (int j = 0; j < FILES; j++) { + lfs_file_close(&lfs, &files[j]); + } + + lfs_unmount(&lfs) => 0; +''' +# TODO FILES=26 found bug +#define.SIZE = [10, 100] +#define.FILES = [4, 10, 26] +define = [ + {SIZE=10, FILES=4}, + {SIZE=10, FILES=10}, + #{SIZE=10, FILES=26}, + {SIZE=100, FILES=4}, + #{SIZE=100, FILES=10}, + #{SIZE=100, FILES=26}, +] +reentrant = true diff --git a/tests_/test_paths.toml b/tests_/test_paths.toml new file mode 100644 index 00000000..480dd9ac --- /dev/null +++ b/tests_/test_paths.toml @@ -0,0 +1,294 @@ + +[[case]] # simple path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "tea") => 0; + lfs_mkdir(&lfs, "tea/hottea") => 0; + lfs_mkdir(&lfs, "tea/warmtea") => 0; + lfs_mkdir(&lfs, "tea/coldtea") => 0; + + lfs_stat(&lfs, "tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "/tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + + lfs_mkdir(&lfs, "/milk") => 0; + lfs_stat(&lfs, "/milk", &info) => 0; + assert(strcmp(info.name, "milk") == 0); + lfs_stat(&lfs, "milk", &info) => 0; + assert(strcmp(info.name, "milk") == 0); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # redundant slashes +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "tea") => 0; + lfs_mkdir(&lfs, "tea/hottea") => 0; + lfs_mkdir(&lfs, "tea/warmtea") => 0; + lfs_mkdir(&lfs, "tea/coldtea") => 0; + + lfs_stat(&lfs, "/tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "//tea//hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "///tea///hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + + lfs_mkdir(&lfs, "////milk") => 0; + lfs_stat(&lfs, "////milk", &info) => 0; + assert(strcmp(info.name, "milk") == 0); + lfs_stat(&lfs, "milk", &info) => 0; + assert(strcmp(info.name, "milk") == 0); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # dot path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "tea") => 0; + lfs_mkdir(&lfs, "tea/hottea") => 0; + lfs_mkdir(&lfs, "tea/warmtea") => 0; + lfs_mkdir(&lfs, "tea/coldtea") => 0; + + lfs_stat(&lfs, "./tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "/./tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "/././tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "/./tea/./hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + + lfs_mkdir(&lfs, "/./milk") => 0; + lfs_stat(&lfs, "/./milk", &info) => 0; + assert(strcmp(info.name, "milk") == 0); + lfs_stat(&lfs, "milk", &info) => 0; + assert(strcmp(info.name, "milk") == 0); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # dot dot path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "tea") => 0; + lfs_mkdir(&lfs, "tea/hottea") => 0; + lfs_mkdir(&lfs, "tea/warmtea") => 0; + lfs_mkdir(&lfs, "tea/coldtea") => 0; + lfs_mkdir(&lfs, "coffee") => 0; + lfs_mkdir(&lfs, "coffee/hotcoffee") => 0; + lfs_mkdir(&lfs, "coffee/warmcoffee") => 0; + lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; + + lfs_stat(&lfs, "coffee/../tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "tea/coldtea/../hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "coffee/coldcoffee/../../tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "coffee/../coffee/../tea/hottea", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + + lfs_mkdir(&lfs, "coffee/../milk") => 0; + lfs_stat(&lfs, "coffee/../milk", &info) => 0; + strcmp(info.name, "milk") => 0; + lfs_stat(&lfs, "milk", &info) => 0; + strcmp(info.name, "milk") => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # trailing dot path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "tea") => 0; + lfs_mkdir(&lfs, "tea/hottea") => 0; + lfs_mkdir(&lfs, "tea/warmtea") => 0; + lfs_mkdir(&lfs, "tea/coldtea") => 0; + + lfs_stat(&lfs, "tea/hottea/", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "tea/hottea/.", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "tea/hottea/./.", &info) => 0; + assert(strcmp(info.name, "hottea") == 0); + lfs_stat(&lfs, "tea/hottea/..", &info) => 0; + assert(strcmp(info.name, "tea") == 0); + lfs_stat(&lfs, "tea/hottea/../.", &info) => 0; + assert(strcmp(info.name, "tea") == 0); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # leading dot path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, ".milk") => 0; + lfs_stat(&lfs, ".milk", &info) => 0; + strcmp(info.name, ".milk") => 0; + lfs_stat(&lfs, "tea/.././.milk", &info) => 0; + strcmp(info.name, ".milk") => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # root dot dot path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "tea") => 0; + lfs_mkdir(&lfs, "tea/hottea") => 0; + lfs_mkdir(&lfs, "tea/warmtea") => 0; + lfs_mkdir(&lfs, "tea/coldtea") => 0; + lfs_mkdir(&lfs, "coffee") => 0; + lfs_mkdir(&lfs, "coffee/hotcoffee") => 0; + lfs_mkdir(&lfs, "coffee/warmcoffee") => 0; + lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; + + lfs_stat(&lfs, "coffee/../../../../../../tea/hottea", &info) => 0; + strcmp(info.name, "hottea") => 0; + + lfs_mkdir(&lfs, "coffee/../../../../../../milk") => 0; + lfs_stat(&lfs, "coffee/../../../../../../milk", &info) => 0; + strcmp(info.name, "milk") => 0; + lfs_stat(&lfs, "milk", &info) => 0; + strcmp(info.name, "milk") => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # invalid path tests +code = ''' + lfs_format(&lfs, &cfg); + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "dirt", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "dirt/ground", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "dirt/ground/earth", &info) => LFS_ERR_NOENT; + + lfs_remove(&lfs, "dirt") => LFS_ERR_NOENT; + lfs_remove(&lfs, "dirt/ground") => LFS_ERR_NOENT; + lfs_remove(&lfs, "dirt/ground/earth") => LFS_ERR_NOENT; + + lfs_mkdir(&lfs, "dirt/ground") => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "dirt/ground", LFS_O_WRONLY | LFS_O_CREAT) + => LFS_ERR_NOENT; + lfs_mkdir(&lfs, "dirt/ground/earth") => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "dirt/ground/earth", LFS_O_WRONLY | LFS_O_CREAT) + => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # root operations +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "/", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_mkdir(&lfs, "/") => LFS_ERR_EXIST; + lfs_file_open(&lfs, &file, "/", LFS_O_WRONLY | LFS_O_CREAT) + => LFS_ERR_ISDIR; + + lfs_remove(&lfs, "/") => LFS_ERR_INVAL; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # root representations +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "/", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_stat(&lfs, "", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_stat(&lfs, ".", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_stat(&lfs, "..", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_stat(&lfs, "//", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_stat(&lfs, "./", &info) => 0; + assert(strcmp(info.name, "/") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_unmount(&lfs) => 0; +''' + +[[case]] # superblock conflict test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "littlefs", &info) => LFS_ERR_NOENT; + lfs_remove(&lfs, "littlefs") => LFS_ERR_NOENT; + + lfs_mkdir(&lfs, "littlefs") => 0; + lfs_stat(&lfs, "littlefs", &info) => 0; + assert(strcmp(info.name, "littlefs") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_remove(&lfs, "littlefs") => 0; + lfs_stat(&lfs, "littlefs", &info) => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # max path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "coffee") => 0; + lfs_mkdir(&lfs, "coffee/hotcoffee") => 0; + lfs_mkdir(&lfs, "coffee/warmcoffee") => 0; + lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; + + memset(path, 'w', LFS_NAME_MAX+1); + path[LFS_NAME_MAX+2] = '\0'; + lfs_mkdir(&lfs, path) => LFS_ERR_NAMETOOLONG; + lfs_file_open(&lfs, &file, path, LFS_O_WRONLY | LFS_O_CREAT) + => LFS_ERR_NAMETOOLONG; + + memcpy(path, "coffee/", strlen("coffee/")); + memset(path+strlen("coffee/"), 'w', LFS_NAME_MAX+1); + path[strlen("coffee/")+LFS_NAME_MAX+2] = '\0'; + lfs_mkdir(&lfs, path) => LFS_ERR_NAMETOOLONG; + lfs_file_open(&lfs, &file, path, LFS_O_WRONLY | LFS_O_CREAT) + => LFS_ERR_NAMETOOLONG; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # really big path test +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "coffee") => 0; + lfs_mkdir(&lfs, "coffee/hotcoffee") => 0; + lfs_mkdir(&lfs, "coffee/warmcoffee") => 0; + lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; + + lfs_mount(&lfs, &cfg) => 0; + memset(path, 'w', LFS_NAME_MAX); + path[LFS_NAME_MAX] = '\0'; + lfs_mkdir(&lfs, path) => 0; + lfs_remove(&lfs, path) => 0; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_remove(&lfs, path) => 0; + + memcpy(path, "coffee/", strlen("coffee/")); + memset(path+strlen("coffee/"), 'w', LFS_NAME_MAX); + path[strlen("coffee/")+LFS_NAME_MAX] = '\0'; + lfs_mkdir(&lfs, path) => 0; + lfs_remove(&lfs, path) => 0; + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_remove(&lfs, path) => 0; + lfs_unmount(&lfs) => 0; +''' + diff --git a/tests_/test_seek.toml b/tests_/test_seek.toml new file mode 100644 index 00000000..0a1d90d1 --- /dev/null +++ b/tests_/test_seek.toml @@ -0,0 +1,380 @@ + +[[case]] # simple file seek +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + size = strlen("kittycatcat"); + memcpy(buffer, "kittycatcat", size); + for (int j = 0; j < COUNT; j++) { + lfs_file_write(&lfs, &file, buffer, size); + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", LFS_O_RDONLY) => 0; + + lfs_soff_t pos; + size = strlen("kittycatcat"); + for (int i = 0; i < SKIP; i++) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + pos = lfs_file_tell(&lfs, &file); + } + pos >= 0 => 1; + + lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_rewind(&lfs, &file) => 0; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, size, LFS_SEEK_CUR) => 3*size; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, -size, LFS_SEEK_CUR) => pos; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, -size, LFS_SEEK_END) >= 0 => 1; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + size = lfs_file_size(&lfs, &file); + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define = [ + {COUNT=132, SKIP=4}, + {COUNT=132, SKIP=128}, + {COUNT=200, SKIP=10}, + {COUNT=200, SKIP=100}, + {COUNT=4, SKIP=1}, + {COUNT=4, SKIP=2}, +] + +[[case]] # simple file seek and write +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + size = strlen("kittycatcat"); + memcpy(buffer, "kittycatcat", size); + for (int j = 0; j < COUNT; j++) { + lfs_file_write(&lfs, &file, buffer, size); + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0; + + lfs_soff_t pos; + size = strlen("kittycatcat"); + for (int i = 0; i < SKIP; i++) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + pos = lfs_file_tell(&lfs, &file); + } + pos >= 0 => 1; + + memcpy(buffer, "doggodogdog", size); + lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; + lfs_file_write(&lfs, &file, buffer, size) => size; + + lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "doggodogdog", size) => 0; + + lfs_file_rewind(&lfs, &file) => 0; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "doggodogdog", size) => 0; + + lfs_file_seek(&lfs, &file, -size, LFS_SEEK_END) >= 0 => 1; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + size = lfs_file_size(&lfs, &file); + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define = [ + {COUNT=132, SKIP=4}, + {COUNT=132, SKIP=128}, + {COUNT=200, SKIP=10}, + {COUNT=200, SKIP=100}, + {COUNT=4, SKIP=1}, + {COUNT=4, SKIP=2}, +] + +[[case]] # boundary seek and writes +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + size = strlen("kittycatcat"); + memcpy(buffer, "kittycatcat", size); + for (int j = 0; j < COUNT; j++) { + lfs_file_write(&lfs, &file, buffer, size); + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0; + + size = strlen("hedgehoghog"); + const lfs_soff_t offsets[] = OFFSETS; + + for (unsigned i = 0; i < sizeof(offsets) / sizeof(offsets[0]); i++) { + lfs_soff_t off = offsets[i]; + memcpy(buffer, "hedgehoghog", size); + lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off; + lfs_file_write(&lfs, &file, buffer, size) => size; + lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hedgehoghog", size) => 0; + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hedgehoghog", size) => 0; + + lfs_file_sync(&lfs, &file) => 0; + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "kittycatcat", size) => 0; + + lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hedgehoghog", size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.COUNT = 132 +define.OFFSETS = '"{512, 1020, 513, 1021, 511, 1019, 1441}"' + +[[case]] # out of bounds seek +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; + size = strlen("kittycatcat"); + memcpy(buffer, "kittycatcat", size); + for (int j = 0; j < COUNT; j++) { + lfs_file_write(&lfs, &file, buffer, size); + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0; + + size = strlen("kittycatcat"); + lfs_file_size(&lfs, &file) => COUNT*size; + lfs_file_seek(&lfs, &file, (COUNT+SKIP)*size, + LFS_SEEK_SET) => (COUNT+SKIP)*size; + lfs_file_read(&lfs, &file, buffer, size) => 0; + + memcpy(buffer, "porcupineee", size); + lfs_file_write(&lfs, &file, buffer, size) => size; + + lfs_file_seek(&lfs, &file, (COUNT+SKIP)*size, + LFS_SEEK_SET) => (COUNT+SKIP)*size; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "porcupineee", size) => 0; + + lfs_file_seek(&lfs, &file, COUNT*size, + LFS_SEEK_SET) => COUNT*size; + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "\0\0\0\0\0\0\0\0\0\0\0", size) => 0; + + lfs_file_seek(&lfs, &file, -((COUNT+SKIP)*size), + LFS_SEEK_CUR) => LFS_ERR_INVAL; + lfs_file_tell(&lfs, &file) => (COUNT+1)*size; + + lfs_file_seek(&lfs, &file, -((COUNT+2*SKIP)*size), + LFS_SEEK_END) => LFS_ERR_INVAL; + lfs_file_tell(&lfs, &file) => (COUNT+1)*size; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define = [ + {COUNT=132, SKIP=4}, + {COUNT=132, SKIP=128}, + {COUNT=200, SKIP=10}, + {COUNT=200, SKIP=100}, + {COUNT=4, SKIP=2}, + {COUNT=4, SKIP=3}, +] + +[[case]] # inline write and seek +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "tinykitty", + LFS_O_RDWR | LFS_O_CREAT) => 0; + int j = 0; + int k = 0; + + memcpy(buffer, "abcdefghijklmnopqrstuvwxyz", 26); + for (unsigned i = 0; i < SIZE; i++) { + lfs_file_write(&lfs, &file, &buffer[j++ % 26], 1) => 1; + lfs_file_tell(&lfs, &file) => i+1; + lfs_file_size(&lfs, &file) => i+1; + } + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; + lfs_file_tell(&lfs, &file) => 0; + lfs_file_size(&lfs, &file) => SIZE; + for (unsigned i = 0; i < SIZE; i++) { + uint8_t c; + lfs_file_read(&lfs, &file, &c, 1) => 1; + c => buffer[k++ % 26]; + } + + lfs_file_sync(&lfs, &file) => 0; + lfs_file_tell(&lfs, &file) => SIZE; + lfs_file_size(&lfs, &file) => SIZE; + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; + for (unsigned i = 0; i < SIZE; i++) { + lfs_file_write(&lfs, &file, &buffer[j++ % 26], 1) => 1; + lfs_file_tell(&lfs, &file) => i+1; + lfs_file_size(&lfs, &file) => SIZE; + lfs_file_sync(&lfs, &file) => 0; + lfs_file_tell(&lfs, &file) => i+1; + lfs_file_size(&lfs, &file) => SIZE; + if (i < SIZE-2) { + uint8_t c[3]; + lfs_file_seek(&lfs, &file, -1, LFS_SEEK_CUR) => i; + lfs_file_read(&lfs, &file, &c, 3) => 3; + lfs_file_tell(&lfs, &file) => i+3; + lfs_file_size(&lfs, &file) => SIZE; + lfs_file_seek(&lfs, &file, i+1, LFS_SEEK_SET) => i+1; + lfs_file_tell(&lfs, &file) => i+1; + lfs_file_size(&lfs, &file) => SIZE; + } + } + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; + lfs_file_tell(&lfs, &file) => 0; + lfs_file_size(&lfs, &file) => SIZE; + for (unsigned i = 0; i < SIZE; i++) { + uint8_t c; + lfs_file_read(&lfs, &file, &c, 1) => 1; + c => buffer[k++ % 26]; + } + + lfs_file_sync(&lfs, &file) => 0; + lfs_file_tell(&lfs, &file) => SIZE; + lfs_file_size(&lfs, &file) => SIZE; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.SIZE = [2, 4, 128, 132] + +[[case]] # file seek and write with power-loss +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + err = lfs_file_open(&lfs, &file, "kitty", LFS_O_RDONLY); + assert(!err || err == LFS_ERR_NOENT); + if (!err) { + if (lfs_file_size(&lfs, &file) != 0) { + lfs_file_size(&lfs, &file) => 11*COUNT; + for (int j = 0; j < COUNT; j++) { + memset(buffer, 0, 11+1); + lfs_file_read(&lfs, &file, buffer, 11) => 11; + assert(memcmp(buffer, "kittycatcat", 11) == 0 || + memcmp(buffer, "doggodogdog", 11) == 0); + } + } + lfs_file_close(&lfs, &file) => 0; + } + + lfs_file_open(&lfs, &file, "kitty", LFS_O_WRONLY | LFS_O_CREAT) => 0; + if (lfs_file_size(&lfs, &file) == 0) { + for (int j = 0; j < COUNT; j++) { + strcpy((char*)buffer, "kittycatcat"); + size = strlen((char*)buffer); + lfs_file_write(&lfs, &file, buffer, size) => size; + } + } + lfs_file_close(&lfs, &file) => 0; + + strcpy((char*)buffer, "doggodogdog"); + size = strlen((char*)buffer); + + lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => COUNT*size; + // seek and write using quadratic probing to touch all + // 11-byte words in the file + lfs_off_t off = 0; + for (int j = 0; j < COUNT; j++) { + off = (5*off + 1) % COUNT; + lfs_file_seek(&lfs, &file, off*size, LFS_SEEK_SET) => off*size; + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, "kittycatcat", size) == 0 || + memcmp(buffer, "doggodogdog", size) == 0); + if (memcmp(buffer, "doggodogdog", size) != 0) { + lfs_file_seek(&lfs, &file, off*size, LFS_SEEK_SET) => off*size; + strcpy((char*)buffer, "doggodogdog"); + lfs_file_write(&lfs, &file, buffer, size) => size; + lfs_file_seek(&lfs, &file, off*size, LFS_SEEK_SET) => off*size; + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, "doggodogdog", size) == 0); + lfs_file_sync(&lfs, &file) => 0; + lfs_file_seek(&lfs, &file, off*size, LFS_SEEK_SET) => off*size; + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, "doggodogdog", size) == 0); + } + } + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => COUNT*size; + for (int j = 0; j < COUNT; j++) { + lfs_file_read(&lfs, &file, buffer, size) => size; + assert(memcmp(buffer, "doggodogdog", size) == 0); + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +# must be power-of-2 for quadratic probing to be exhaustive +define.COUNT = [4, 64, 128] +reentrant = true diff --git a/tests_/test_truncate.toml b/tests_/test_truncate.toml new file mode 100644 index 00000000..fde2da51 --- /dev/null +++ b/tests_/test_truncate.toml @@ -0,0 +1,395 @@ +[[case]] # simple truncate +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldynoop", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + strcpy((char*)buffer, "hair"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < LARGESIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => LARGESIZE; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldynoop", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => LARGESIZE; + + lfs_file_truncate(&lfs, &file, MEDIUMSIZE) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldynoop", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + size = strlen("hair"); + for (lfs_off_t j = 0; j < MEDIUMSIZE; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hair", size) => 0; + } + lfs_file_read(&lfs, &file, buffer, size) => 0; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.MEDIUMSIZE = [32, 2048] +define.LARGESIZE = 8192 + +[[case]] # truncate and read +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldyread", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + strcpy((char*)buffer, "hair"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < LARGESIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => LARGESIZE; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldyread", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => LARGESIZE; + + lfs_file_truncate(&lfs, &file, MEDIUMSIZE) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + size = strlen("hair"); + for (lfs_off_t j = 0; j < MEDIUMSIZE; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hair", size) => 0; + } + lfs_file_read(&lfs, &file, buffer, size) => 0; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldyread", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + size = strlen("hair"); + for (lfs_off_t j = 0; j < MEDIUMSIZE; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hair", size) => 0; + } + lfs_file_read(&lfs, &file, buffer, size) => 0; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.MEDIUMSIZE = [32, 2048] +define.LARGESIZE = 8192 + +[[case]] # write, truncate, and read +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "sequence", + LFS_O_RDWR | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + size = lfs.cfg->cache_size; + lfs_size_t qsize = size / 4; + uint8_t *wb = buffer; + uint8_t *rb = buffer + size; + for (lfs_off_t j = 0; j < size; ++j) { + wb[j] = j; + } + + /* Spread sequence over size */ + lfs_file_write(&lfs, &file, wb, size) => size; + lfs_file_size(&lfs, &file) => size; + lfs_file_tell(&lfs, &file) => size; + + lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; + lfs_file_tell(&lfs, &file) => 0; + + /* Chop off the last quarter */ + lfs_size_t trunc = size - qsize; + lfs_file_truncate(&lfs, &file, trunc) => 0; + lfs_file_tell(&lfs, &file) => 0; + lfs_file_size(&lfs, &file) => trunc; + + /* Read should produce first 3/4 */ + lfs_file_read(&lfs, &file, rb, size) => trunc; + memcmp(rb, wb, trunc) => 0; + + /* Move to 1/4 */ + lfs_file_size(&lfs, &file) => trunc; + lfs_file_seek(&lfs, &file, qsize, LFS_SEEK_SET) => qsize; + lfs_file_tell(&lfs, &file) => qsize; + + /* Chop to 1/2 */ + trunc -= qsize; + lfs_file_truncate(&lfs, &file, trunc) => 0; + lfs_file_tell(&lfs, &file) => qsize; + lfs_file_size(&lfs, &file) => trunc; + + /* Read should produce second quarter */ + lfs_file_read(&lfs, &file, rb, size) => trunc - qsize; + memcmp(rb, wb + qsize, trunc - qsize) => 0; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # truncate and write +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldywrite", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + strcpy((char*)buffer, "hair"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < LARGESIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => LARGESIZE; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldywrite", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => LARGESIZE; + + lfs_file_truncate(&lfs, &file, MEDIUMSIZE) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + strcpy((char*)buffer, "bald"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < MEDIUMSIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "baldywrite", LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + + size = strlen("bald"); + for (lfs_off_t j = 0; j < MEDIUMSIZE; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "bald", size) => 0; + } + lfs_file_read(&lfs, &file, buffer, size) => 0; + + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' +define.MEDIUMSIZE = [32, 2048] +define.LARGESIZE = 8192 + +[[case]] # truncate write under powerloss +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + err = lfs_file_open(&lfs, &file, "baldy", LFS_O_RDONLY); + assert(!err || err == LFS_ERR_NOENT); + if (!err) { + size = lfs_file_size(&lfs, &file); + assert(size == 0 || + size == LARGESIZE || + size == MEDIUMSIZE || + size == SMALLSIZE); + for (lfs_off_t j = 0; j < size; j += 4) { + lfs_file_read(&lfs, &file, buffer, 4) => 4; + assert(memcmp(buffer, "hair", 4) == 0 || + memcmp(buffer, "bald", 4) == 0 || + memcmp(buffer, "comb", 4) == 0); + } + lfs_file_close(&lfs, &file) => 0; + } + + lfs_file_open(&lfs, &file, "baldy", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + lfs_file_size(&lfs, &file) => 0; + strcpy((char*)buffer, "hair"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < LARGESIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => LARGESIZE; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "baldy", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => LARGESIZE; + lfs_file_truncate(&lfs, &file, MEDIUMSIZE) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + strcpy((char*)buffer, "bald"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < MEDIUMSIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "baldy", LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => MEDIUMSIZE; + lfs_file_truncate(&lfs, &file, SMALLSIZE) => 0; + lfs_file_size(&lfs, &file) => SMALLSIZE; + strcpy((char*)buffer, "comb"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < SMALLSIZE; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => SMALLSIZE; + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; +''' +define.SMALLSIZE = [4, 512] +define.MEDIUMSIZE = [32, 1024] +define.LARGESIZE = 2048 +reentrant = true + +[[case]] # more aggressive general truncation tests +code = ''' + #define COUNT 5 + const struct { + lfs_off_t startsizes[COUNT]; + lfs_off_t startseeks[COUNT]; + lfs_off_t hotsizes[COUNT]; + lfs_off_t coldsizes[COUNT]; + } configs[] = { + // cold shrinking + {{2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + { 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}}, + // cold expanding + {{2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + { 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}}, + // warm shrinking truncate + {{2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + { 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}, + { 0, 0, 0, 0, 0}}, + // warm expanding truncate + {{ 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}, + { 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}}, + // mid-file shrinking truncate + {{2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + { LARGESIZE, LARGESIZE, LARGESIZE, LARGESIZE, LARGESIZE}, + { 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}, + { 0, 0, 0, 0, 0}}, + // mid-file expanding truncate + {{ 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE, 2*LARGESIZE}, + { 0, 0, SMALLSIZE, MEDIUMSIZE, LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}, + {2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE, 2*LARGESIZE}}, + }; + + const lfs_off_t *startsizes = configs[CONFIG].startsizes; + const lfs_off_t *startseeks = configs[CONFIG].startseeks; + const lfs_off_t *hotsizes = configs[CONFIG].hotsizes; + const lfs_off_t *coldsizes = configs[CONFIG].coldsizes; + + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + for (unsigned i = 0; i < COUNT; i++) { + sprintf(path, "hairyhead%d", i); + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + strcpy((char*)buffer, "hair"); + size = strlen((char*)buffer); + for (lfs_off_t j = 0; j < startsizes[i]; j += size) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_size(&lfs, &file) => startsizes[i]; + + if (startseeks[i] != startsizes[i]) { + lfs_file_seek(&lfs, &file, + startseeks[i], LFS_SEEK_SET) => startseeks[i]; + } + + lfs_file_truncate(&lfs, &file, hotsizes[i]) => 0; + lfs_file_size(&lfs, &file) => hotsizes[i]; + + lfs_file_close(&lfs, &file) => 0; + } + + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + + for (unsigned i = 0; i < COUNT; i++) { + sprintf(path, "hairyhead%d", i); + lfs_file_open(&lfs, &file, path, LFS_O_RDWR) => 0; + lfs_file_size(&lfs, &file) => hotsizes[i]; + + size = strlen("hair"); + lfs_off_t j = 0; + for (; j < startsizes[i] && j < hotsizes[i]; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hair", size) => 0; + } + + for (; j < hotsizes[i]; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "\0\0\0\0", size) => 0; + } + + lfs_file_truncate(&lfs, &file, coldsizes[i]) => 0; + lfs_file_size(&lfs, &file) => coldsizes[i]; + + lfs_file_close(&lfs, &file) => 0; + } + + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + + for (unsigned i = 0; i < COUNT; i++) { + sprintf(path, "hairyhead%d", i); + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + lfs_file_size(&lfs, &file) => coldsizes[i]; + + size = strlen("hair"); + lfs_off_t j = 0; + for (; j < startsizes[i] && j < hotsizes[i] && j < coldsizes[i]; + j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "hair", size) => 0; + } + + for (; j < coldsizes[i]; j += size) { + lfs_file_read(&lfs, &file, buffer, size) => size; + memcmp(buffer, "\0\0\0\0", size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + } + + lfs_unmount(&lfs) => 0; +''' +define.CONFIG = 'range(6)' +define.SMALLSIZE = 32 +define.MEDIUMSIZE = 2048 +define.LARGESIZE = 8192 + From 5181ce66cd704b3420fd260943fa07b2ddf7e8ff Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 14 Jan 2020 09:14:01 -0600 Subject: [PATCH 07/41] Migrated the first of the tests with internal knowledge Both test_move and test_orphan needed internal knowledge which comes with the addition of the "in" attribute. This was in the plan for the test-revamp from the beginning as it really opens up the ability to write more unit-style-tests using internal knowledge of how littlefs works. More unit-style-tests should help _fix_ bugs by limiting the scope of the test and where the bug could be hiding. The "in" attribute effectively runs tests _inside_ the .c file specified, giving the test access to all static members without needed to change their visibility. --- scripts/test_.py | 234 +++++---- tests_/test_alloc.toml | 38 +- tests_/test_dirs.toml | 26 +- tests_/test_files.toml | 50 +- tests_/test_format.toml | 16 +- tests_/test_interspersed.toml | 52 +- tests_/test_move.toml | 957 ++++++++++++++++++++++++++++++++++ tests_/test_orphan.toml | 56 ++ tests_/test_seek.toml | 48 +- tests_/test_truncate.toml | 29 +- 10 files changed, 1271 insertions(+), 235 deletions(-) create mode 100644 tests_/test_move.toml create mode 100644 tests_/test_orphan.toml diff --git a/scripts/test_.py b/scripts/test_.py index ab674c97..760f64e7 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -5,18 +5,20 @@ # # TODO -# - nargs > 1? +# x nargs > 1? # x show perm config on failure # x filtering # n show perm config on verbose? -# - better lineno tracking for cases? +# x better lineno tracking for cases? # n non-int perms? -# - different path format? +# x different path format? # - suite.prologue, suite.epilogue -# - in +# x in # x change BLOCK_CYCLES to -1 by default # x change persist behaviour # x config chaining correct +# - why can't gdb see my defines? +# - say no to internal? import toml import glob @@ -34,17 +36,14 @@ TESTDIR = 'tests_' RULES = """ define FLATTEN -%$(subst /,.,$(target:.c=.tc)): $(target) - cat <(echo '#line 1 "$$<"') $$< > $$@ +tests_/%$(subst /,.,$(target)): $(target) + ./scripts/explode_asserts.py $$< -o $$@ endef $(foreach target,$(SRC),$(eval $(FLATTEN))) -include tests_/*.d .SECONDARY: -%.c: %.tc - ./scripts/explode_asserts.py $< -o $@ - %.test: override CFLAGS += -fdiagnostics-color=always %.test: override CFLAGS += -ggdb %.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.$f) @@ -56,7 +55,6 @@ #include "filebd/lfs_filebd.h" #include "rambd/lfs_rambd.h" #include -const char *LFS_DISK = NULL; """ DEFINES = { "LFS_READ_SIZE": 16, @@ -70,6 +68,8 @@ } PROLOGUE = """ // prologue + extern const char *LFS_DISK; + __attribute__((unused)) lfs_t lfs; __attribute__((unused)) lfs_filebd_t filebd; __attribute__((unused)) lfs_rambd_t rambd; @@ -129,28 +129,31 @@ def __init__(self, case, returncode=None, stdout=None, assert_=None): self.assert_ = assert_ class TestCase: - def __init__(self, config, suite=None, caseno=None, lineno=None, **_): + def __init__(self, config, filter=filter, + suite=None, caseno=None, lineno=None, **_): + self.filter = filter self.suite = suite self.caseno = caseno self.lineno = lineno self.code = config['code'] + self.code_lineno = config['code_lineno'] self.defines = config.get('define', {}) self.if_ = config.get('if', None) - self.leaky = config.get('leaky', False) + self.in_ = config.get('in', None) def __str__(self): if hasattr(self, 'permno'): if any(k not in self.case.defines for k in self.defines): - return '%s[%d,%d] (%s)' % ( + return '%s#%d#%d (%s)' % ( self.suite.name, self.caseno, self.permno, ', '.join( '%s=%s' % (k, v) for k, v in self.defines.items() if k not in self.case.defines)) else: - return '%s[%d,%d]' % ( + return '%s#%d#%d' % ( self.suite.name, self.caseno, self.permno) else: - return '%s[%d]' % ( + return '%s#%d' % ( self.suite.name, self.caseno) def permute(self, defines, permno=None, **_): @@ -163,17 +166,10 @@ def permute(self, defines, permno=None, **_): def build(self, f, **_): # prologue - f.write('void test_case%d(' % self.caseno) - first = True - for k, v in sorted(self.perms[0].defines.items()): - if k not in self.defines: - if not first: - f.write(',') - else: - first = False - f.write('\n') - f.write(8*' '+'__attribute__((unused)) intmax_t %s' % k) - f.write(') {\n') + f.write('void test_case%d(%s) {\n' % (self.caseno, ','.join( + '\n'+8*' '+'__attribute__((unused)) intmax_t %s' % k + for k in sorted(self.perms[0].defines) + if k not in self.defines))) for k, v in sorted(self.defines.items()): if k not in self.suite.defines: @@ -182,7 +178,7 @@ def build(self, f, **_): f.write(PROLOGUE) f.write('\n') f.write(4*' '+'// test case %d\n' % self.caseno) - f.write(4*' '+'#line %d "%s"\n' % (self.lineno, self.suite.path)) + f.write(4*' '+'#line %d "%s"\n' % (self.code_lineno, self.suite.path)) # test case goes here f.write(self.code) @@ -198,7 +194,15 @@ def build(self, f, **_): f.write('}\n') def shouldtest(self, **args): - if self.if_ is not None: + if (self.filter is not None and + len(self.filter) >= 1 and + self.filter[0] != self.caseno): + return False + elif (self.filter is not None and + len(self.filter) >= 2 and + self.filter[1] != self.permno): + return False + elif self.if_ is not None: return eval(self.if_, None, self.defines.copy()) else: return True @@ -227,7 +231,7 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): ncmd.extend(['-ex', 'up']) elif gdb == 'start': ncmd.extend([ - '-ex', 'b %s:%d' % (self.suite.path, self.lineno), + '-ex', 'b %s:%d' % (self.suite.path, self.code_lineno), '-ex', 'r']) ncmd.extend(['--args'] + cmd) @@ -309,7 +313,7 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): for cycles in it.count(1): # exact cycle we should drop into debugger? if gdb and failure and failure.cycleno == cycles: - return super().test(exec=exec, persist=True, + return super().test(exec=exec, persist='noerase', gdb=gdb, failure=failure, **args) # run tests, but kill the program after prog/erase has @@ -337,11 +341,12 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): raise class TestSuite: - def __init__(self, path, TestCase=TestCase, **args): + def __init__(self, path, filter=None, TestCase=TestCase, **args): self.name = os.path.basename(path) if self.name.endswith('.toml'): self.name = self.name[:-len('.toml')] self.path = path + self.filter = filter self.TestCase = TestCase with open(path) as f: @@ -351,9 +356,12 @@ def __init__(self, path, TestCase=TestCase, **args): # find line numbers f.seek(0) linenos = [] + code_linenos = [] for i, line in enumerate(f): - if re.match(r'^\s*code\s*=\s*(\'\'\'|""")', line): - linenos.append(i + 2) + if re.match(r'\[\[\s*case\s*\]\]', line): + linenos.append(i+1) + if re.match(r'code\s*=\s*(\'\'\'|""")', line): + code_linenos.append(i+2) # grab global config self.defines = config.get('define', {}) @@ -361,12 +369,15 @@ def __init__(self, path, TestCase=TestCase, **args): # create initial test cases self.cases = [] for i, (case, lineno) in enumerate(zip(config['case'], linenos)): + # code lineno? + if 'code' in case: + case['code_lineno'] = code_linenos.pop(0) # give our case's config a copy of our "global" config for k, v in config.items(): if k not in case: case[k] = v # initialize test case - self.cases.append(self.TestCase(case, + self.cases.append(self.TestCase(case, filter=filter, suite=self, caseno=i, lineno=lineno, **args)) def __str__(self): @@ -438,40 +449,52 @@ def permute(self, defines={}, **args): return self.perms def build(self, **args): - # build test.c - f = io.StringIO() - f.write(GLOBALS) + # build test files + tf = open(self.path + '.test.c.t', 'w') + tf.write(GLOBALS) + tfs = {None: tf} for case in self.cases: - f.write('\n') - case.build(f, **args) - - f.write('\n') - f.write('int main(int argc, char **argv) {\n') - f.write(4*' '+'int case_ = (argc >= 2) ? atoi(argv[1]) : 0;\n') - f.write(4*' '+'int perm = (argc >= 3) ? atoi(argv[2]) : 0;\n') - f.write(4*' '+'LFS_DISK = (argc >= 4) ? argv[3] : NULL;\n') + if case.in_ not in tfs: + tfs[case.in_] = open(self.path+'.'+ + case.in_.replace('/', '.')+'.t', 'w') + tfs[case.in_].write('#line 1 "%s"\n' % case.in_) + with open(case.in_) as f: + for line in f: + tfs[case.in_].write(line) + tfs[case.in_].write('\n') + tfs[case.in_].write(GLOBALS) + + tfs[case.in_].write('\n') + case.build(tfs[case.in_], **args) + + tf.write('\n') + tf.write('const char *LFS_DISK = NULL;\n') + tf.write('int main(int argc, char **argv) {\n') + tf.write(4*' '+'int case_ = (argc >= 2) ? atoi(argv[1]) : 0;\n') + tf.write(4*' '+'int perm = (argc >= 3) ? atoi(argv[2]) : 0;\n') + tf.write(4*' '+'LFS_DISK = (argc >= 4) ? argv[3] : NULL;\n') for perm in self.perms: - f.write(4*' '+'if (argc < 3 || ' - '(case_ == %d && perm == %d)) { ' % ( - perm.caseno, perm.permno)) - f.write('test_case%d(' % perm.caseno) - first = True - for k, v in sorted(perm.defines.items()): - if k not in perm.case.defines: - if not first: - f.write(', ') - else: - first = False - f.write(str(v)) - f.write('); }\n') - f.write('}\n') - - # add test-related rules - rules = RULES.replace(4*' ', '\t') - + # test declaration + tf.write(4*' '+'extern void test_case%d(%s);\n' % ( + perm.caseno, ', '.join( + 'intmax_t %s' % k for k in sorted(perm.defines) + if k not in perm.case.defines))) + # test call + tf.write(4*' '+ + 'if (argc < 3 || (case_ == %d && perm == %d)) {' + ' test_case%d(%s); ' + '}\n' % (perm.caseno, perm.permno, perm.caseno, ', '.join( + str(v) for k, v in sorted(perm.defines.items()) + if k not in perm.case.defines))) + tf.write('}\n') + + for tf in tfs.values(): + tf.close() + + # write makefiles with open(self.path + '.mk', 'w') as mk: - mk.write(rules) + mk.write(RULES.replace(4*' ', '\t')) mk.write('\n') # add truely global defines globally @@ -479,12 +502,18 @@ def build(self, **args): mk.write('%s: override CFLAGS += -D%s=%r\n' % ( self.path+'.test', k, v)) - # write test.c in base64 so make can decide when to rebuild - mk.write('%s: %s\n' % (self.path+'.test.tc', self.path)) - mk.write('\t@base64 -d <<< ') - mk.write(base64.b64encode( - f.getvalue().encode('utf8')).decode('utf8')) - mk.write(' > $@\n') + for path in tfs: + if path is None: + mk.write('%s: %s | %s\n' % ( + self.path+'.test.c', + self.path, + self.path+'.test.c.t')) + else: + mk.write('%s: %s %s | %s\n' % ( + self.path+'.'+path.replace('/', '.'), + self.path, path, + self.path+'.'+path.replace('/', '.')+'.t')) + mk.write('\t./scripts/explode_asserts.py $| -o $@\n') self.makefile = self.path + '.mk' self.target = self.path + '.test' @@ -524,37 +553,33 @@ def test(self, caseno=None, permno=None, **args): sys.stdout.write('\n') def main(**args): - testpath = args['testpath'] - - # optional brackets for specific test - m = re.search(r'\[(\d+)(?:,(\d+))?\]$', testpath) - if m: - caseno = int(m.group(1)) - permno = int(m.group(2)) if m.group(2) is not None else None - testpath = testpath[:m.start()] - else: - caseno = None - permno = None - - # figure out the suite's toml file - if os.path.isdir(testpath): - testpath = testpath + '/test_*.toml' - elif os.path.isfile(testpath): - testpath = testpath - elif testpath.endswith('.toml'): - testpath = TESTDIR + '/' + testpath - else: - testpath = TESTDIR + '/' + testpath + '.toml' - - # find tests suites = [] - for path in glob.glob(testpath): - if args.get('valgrind', False): - suites.append(TestSuite(path, TestCase=ValgrindTestCase, **args)) - elif args.get('reentrant', False): - suites.append(TestSuite(path, TestCase=ReentrantTestCase, **args)) + for testpath in args['testpaths']: + # optionally specified test case/perm + testpath, *filter = testpath.split('#') + filter = [int(f) for f in filter] + + # figure out the suite's toml file + if os.path.isdir(testpath): + testpath = testpath + '/test_*.toml' + elif os.path.isfile(testpath): + testpath = testpath + elif testpath.endswith('.toml'): + testpath = TESTDIR + '/' + testpath else: - suites.append(TestSuite(path, **args)) + testpath = TESTDIR + '/' + testpath + '.toml' + + # find tests + for path in glob.glob(testpath): + if args.get('valgrind', False): + TestCase_ = ValgrindTestCase + elif args.get('reentrant', False): + TestCase_ = ReentrantTestCase + else: + TestCase_ = TestCase + + suites.append(TestSuite(path, + filter=filter, TestCase=TestCase_, **args)) # sort for reproducability suites = sorted(suites) @@ -632,7 +657,7 @@ def main(**args): print('====== testing ======') try: for suite in suites: - suite.test(caseno, permno, **args) + suite.test(**args) except TestFailure: pass @@ -647,11 +672,10 @@ def main(**args): if perm.result == PASS: passed += 1 else: - #sys.stdout.write("--- %s ---\n" % perm) sys.stdout.write( "\033[01m{path}:{lineno}:\033[01;31mfailure:\033[m " "{perm} failed with {returncode}\n".format( - perm=perm, path=perm.suite.path, lineno=perm.lineno-2, + perm=perm, path=perm.suite.path, lineno=perm.lineno, returncode=perm.result.returncode or 0)) if perm.result.stdout: for line in (perm.result.stdout @@ -681,15 +705,15 @@ def main(**args): failure.case.test(failure=failure, **args) sys.exit(0) - print('tests passed: %d' % passed) print('tests failed: %d' % failed) + return 1 if failed > 0 else 0 if __name__ == "__main__": import argparse parser = argparse.ArgumentParser( description="Run parameterized tests in various configurations.") - parser.add_argument('testpath', nargs='?', default=TESTDIR, + parser.add_argument('testpaths', nargs='*', default=[TESTDIR], help="Description of test(s) to run. By default, this is all tests \ found in the \"{0}\" directory. Here, you can specify a different \ directory of tests, a specific file, a suite by name, and even a \ @@ -713,4 +737,4 @@ def main(**args): help="Run reentrant tests with simulated power-loss.") parser.add_argument('-e', '--exec', default=[], type=lambda e: e.split(' '), help="Run tests with another executable prefixed on the command line.") - main(**vars(parser.parse_args())) + sys.exit(main(**vars(parser.parse_args()))) diff --git a/tests_/test_alloc.toml b/tests_/test_alloc.toml index 6851e772..ce80411a 100644 --- a/tests_/test_alloc.toml +++ b/tests_/test_alloc.toml @@ -2,6 +2,8 @@ # note for these to work there are many constraints on the device geometry [[case]] # parallel allocation test +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; lfs_file_t files[FILES]; @@ -41,10 +43,10 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' -define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' [[case]] # serial allocation test +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; @@ -80,10 +82,11 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' -define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' [[case]] # parallel allocation reuse test +define.FILES = 3 +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.CYCLES = [1, 10] code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; lfs_file_t files[FILES]; @@ -134,11 +137,11 @@ code = ''' lfs_unmount(&lfs) => 0; } ''' + +[[case]] # serial allocation reuse test define.FILES = 3 define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' define.CYCLES = [1, 10] - -[[case]] # serial allocation reuse test code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; @@ -185,9 +188,6 @@ code = ''' lfs_unmount(&lfs) => 0; } ''' -define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' -define.CYCLES = [1, 10] [[case]] # exhaustion test code = ''' @@ -226,6 +226,7 @@ code = ''' ''' [[case]] # exhaustion wraparound test +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / 3)' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -271,7 +272,6 @@ code = ''' lfs_remove(&lfs, "exhaustion") => 0; lfs_unmount(&lfs) => 0; ''' -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / 3)' [[case]] # dir exhaustion test code = ''' @@ -327,6 +327,9 @@ code = ''' # should be removed and replaced with generalized tests. [[case]] # chained dir exhaustion test +define.LFS_BLOCK_SIZE = 512 +define.LFS_BLOCK_COUNT = 1024 +if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -393,11 +396,11 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' + +[[case]] # split dir test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' - -[[case]] # split dir test code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -438,11 +441,11 @@ code = ''' lfs_unmount(&lfs) => 0; ''' + +[[case]] # outdated lookahead test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' - -[[case]] # outdated lookahead test code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -501,11 +504,11 @@ code = ''' } lfs_file_close(&lfs, &file) => 0; ''' + +[[case]] # outdated lookahead and split dir test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' - -[[case]] # outdated lookahead and split dir test code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -561,6 +564,3 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -define.LFS_BLOCK_SIZE = 512 -define.LFS_BLOCK_COUNT = 1024 -if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' diff --git a/tests_/test_dirs.toml b/tests_/test_dirs.toml index 65987538..712f24ff 100644 --- a/tests_/test_dirs.toml +++ b/tests_/test_dirs.toml @@ -15,6 +15,7 @@ code = ''' ''' [[case]] # many directory creation +define.N = 'range(0, 100, 3)' code = ''' lfs_format(&lfs, &cfg) => 0; @@ -43,9 +44,9 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; ''' -define.N = 'range(0, 100, 3)' [[case]] # many directory removal +define.N = 'range(3, 100, 11)' code = ''' lfs_format(&lfs, &cfg) => 0; @@ -93,9 +94,9 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; ''' -define.N = 'range(3, 100, 11)' [[case]] # many directory rename +define.N = 'range(3, 100, 11)' code = ''' lfs_format(&lfs, &cfg) => 0; @@ -152,9 +153,10 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs); ''' -define.N = 'range(3, 100, 11)' [[case]] # reentrant many directory creation/rename/removal +define.N = [5, 25] +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -231,10 +233,9 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; ''' -define.N = [5, 25] -reentrant = true [[case]] # file creation +define.N = 'range(3, 100, 11)' code = ''' lfs_format(&lfs, &cfg) => 0; @@ -265,9 +266,9 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs); ''' -define.N = 'range(3, 100, 11)' [[case]] # file removal +define.N = 'range(0, 100, 3)' code = ''' lfs_format(&lfs, &cfg) => 0; @@ -317,9 +318,9 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; ''' -define.N = 'range(0, 100, 3)' [[case]] # file rename +define.N = 'range(0, 100, 3)' code = ''' lfs_format(&lfs, &cfg) => 0; @@ -378,9 +379,10 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs); ''' -define.N = 'range(0, 100, 3)' [[case]] # reentrant file creation/rename/removal +define.N = [5, 25] +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -457,8 +459,6 @@ code = ''' lfs_dir_close(&lfs, &dir) => 0; lfs_unmount(&lfs) => 0; ''' -define.N = [5, 25] -reentrant = true [[case]] # nested directories code = ''' @@ -585,6 +585,7 @@ code = ''' ''' [[case]] # recursive remove +define.N = [10, 100] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -639,7 +640,6 @@ code = ''' lfs_remove(&lfs, "prickly-pear") => LFS_ERR_NOENT; lfs_unmount(&lfs) => 0; ''' -define.N = [10, 100] [[case]] # other error cases code = ''' @@ -716,6 +716,7 @@ code = ''' ''' [[case]] # directory seek +define.COUNT = [4, 128, 132] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -774,9 +775,9 @@ code = ''' lfs_unmount(&lfs) => 0; } ''' -define.COUNT = [4, 128, 132] [[case]] # root seek +define.COUNT = [4, 128, 132] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -834,5 +835,4 @@ code = ''' lfs_unmount(&lfs) => 0; } ''' -define.COUNT = [4, 128, 132] diff --git a/tests_/test_files.toml b/tests_/test_files.toml index ce161786..da132f26 100644 --- a/tests_/test_files.toml +++ b/tests_/test_files.toml @@ -20,6 +20,8 @@ code = ''' ''' [[case]] # larger files +define.SIZE = [32, 8192, 262144, 0, 7, 8193] +define.CHUNKSIZE = [31, 16, 33, 1, 1023] code = ''' lfs_format(&lfs, &cfg) => 0; @@ -54,10 +56,11 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.SIZE = [32, 8192, 262144, 0, 7, 8193] -define.CHUNKSIZE = [31, 16, 33, 1, 1023] [[case]] # rewriting files +define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] +define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] +define.CHUNKSIZE = [31, 16, 1, 1025] code = ''' lfs_format(&lfs, &cfg) => 0; @@ -135,11 +138,11 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' + +[[case]] # appending files define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] define.CHUNKSIZE = [31, 16, 1, 1025] - -[[case]] # appending files code = ''' lfs_format(&lfs, &cfg) => 0; @@ -212,11 +215,11 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' + +[[case]] # truncating files define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] define.CHUNKSIZE = [31, 16, 1, 1025] - -[[case]] # truncating files code = ''' lfs_format(&lfs, &cfg) => 0; @@ -281,11 +284,11 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] -define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] -define.CHUNKSIZE = [31, 16, 1, 1025] [[case]] # reentrant file writing +define.SIZE = [32, 0, 7, 2049] +define.CHUNKSIZE = [31, 16, 65] +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -329,11 +332,17 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.SIZE = [32, 0, 7, 2049] -define.CHUNKSIZE = [31, 16, 65] -reentrant = true [[case]] # reentrant file writing with syncs +define = [ + # append (O(n)) + {MODE='LFS_O_APPEND', SIZE=[32, 0, 7, 2049], CHUNKSIZE=[31, 16, 65]}, + # truncate (O(n^2)) + {MODE='LFS_O_TRUNC', SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]}, + # rewrite (O(n^2)) + {MODE=0, SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]}, +] +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -393,17 +402,9 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define = [ - # append (O(n)) - {MODE='LFS_O_APPEND', SIZE=[32, 0, 7, 2049], CHUNKSIZE=[31, 16, 65]}, - # truncate (O(n^2)) - {MODE='LFS_O_TRUNC', SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]}, - # rewrite (O(n^2)) - {MODE=0, SIZE=[32, 0, 7, 200], CHUNKSIZE=[31, 16, 65]}, -] -reentrant = true [[case]] # many files +define.N = 300 code = ''' lfs_format(&lfs, &cfg) => 0; // create N files of 7 bytes @@ -426,9 +427,9 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' -define.N = 300 [[case]] # many files with power cycle +define.N = 300 code = ''' lfs_format(&lfs, &cfg) => 0; // create N files of 7 bytes @@ -453,9 +454,10 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' -define.N = 300 [[case]] # many files with power loss +define.N = 300 +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -482,5 +484,3 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' -define.N = 300 -reentrant = true diff --git a/tests_/test_format.toml b/tests_/test_format.toml index c52db2a4..702c0393 100644 --- a/tests_/test_format.toml +++ b/tests_/test_format.toml @@ -11,6 +11,7 @@ code = ''' ''' [[case]] # reentrant format +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -19,7 +20,6 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' -reentrant = true [[case]] # invalid mount code = ''' @@ -29,6 +29,8 @@ code = ''' # TODO invalid superblock? (corrupt 1, 0) [[case]] # expanding superblock +define.BLOCK_CYCLES = [32, 33, 1] +define.N = [10, 100, 1000] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -47,10 +49,10 @@ code = ''' assert(strcmp(info.name, "dummy") == 0); lfs_unmount(&lfs) => 0; ''' -define.BLOCK_CYCLES = [32, 33, 1] -define.N = [10, 100, 1000] [[case]] # expanding superblock with power cycle +define.BLOCK_CYCLES = [32, 33, 1] +define.N = [10, 100, 1000] code = ''' lfs_format(&lfs, &cfg) => 0; for (int i = 0; i < N; i++) { @@ -71,10 +73,11 @@ code = ''' assert(strcmp(info.name, "dummy") == 0); lfs_unmount(&lfs) => 0; ''' -define.BLOCK_CYCLES = [32, 33, 1] -define.N = [10, 100, 1000] [[case]] # reentrant expanding superblock +define.BLOCK_CYCLES = [2, 1] +define.N = 24 +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -100,6 +103,3 @@ code = ''' assert(strcmp(info.name, "dummy") == 0); lfs_unmount(&lfs) => 0; ''' -define.BLOCK_CYCLES = [2, 1] -define.N = 24 -reentrant = true diff --git a/tests_/test_interspersed.toml b/tests_/test_interspersed.toml index ef4f9d24..32f79e79 100644 --- a/tests_/test_interspersed.toml +++ b/tests_/test_interspersed.toml @@ -1,5 +1,16 @@ [[case]] # interspersed file test +# TODO FILES=26 found bug +#define.SIZE = [10, 100] +#define.FILES = [4, 10, 26] +define = [ + {SIZE=10, FILES=4}, + {SIZE=10, FILES=10}, + #{SIZE=10, FILES=26}, + {SIZE=100, FILES=4}, + {SIZE=100, FILES=10}, + #{SIZE=100, FILES=26}, +] code = ''' lfs_file_t files[FILES]; const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; @@ -56,19 +67,10 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -# TODO FILES=26 found bug -#define.SIZE = [10, 100] -#define.FILES = [4, 10, 26] -define = [ - {SIZE=10, FILES=4}, - {SIZE=10, FILES=10}, - #{SIZE=10, FILES=26}, - {SIZE=100, FILES=4}, - {SIZE=100, FILES=10}, - #{SIZE=100, FILES=26}, -] [[case]] # interspersed remove file test +define.SIZE = [10, 100] +define.FILES = [4, 10, 26] code = ''' const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; lfs_format(&lfs, &cfg) => 0; @@ -118,10 +120,9 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -define.SIZE = [10, 100] -define.FILES = [4, 10, 26] [[case]] # remove inconveniently test +define.SIZE = [10, 100] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -179,9 +180,20 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -define.SIZE = [10, 100] [[case]] # reentrant interspersed file test +# TODO FILES=26 found bug +#define.SIZE = [10, 100] +#define.FILES = [4, 10, 26] +define = [ + {SIZE=10, FILES=4}, + {SIZE=10, FILES=10}, + #{SIZE=10, FILES=26}, + {SIZE=100, FILES=4}, + #{SIZE=100, FILES=10}, + #{SIZE=100, FILES=26}, +] +reentrant = true code = ''' lfs_file_t files[FILES]; const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; @@ -248,15 +260,3 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -# TODO FILES=26 found bug -#define.SIZE = [10, 100] -#define.FILES = [4, 10, 26] -define = [ - {SIZE=10, FILES=4}, - {SIZE=10, FILES=10}, - #{SIZE=10, FILES=26}, - {SIZE=100, FILES=4}, - #{SIZE=100, FILES=10}, - #{SIZE=100, FILES=26}, -] -reentrant = true diff --git a/tests_/test_move.toml b/tests_/test_move.toml new file mode 100644 index 00000000..c49b0384 --- /dev/null +++ b/tests_/test_move.toml @@ -0,0 +1,957 @@ +[[case]] # move file +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hola\n", 5) => 5; + lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; + lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hello", "c/hello") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move file corrupt source +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hola\n", 5) => 5; + lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; + lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hello", "c/hello") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the source + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move file corrupt source and dest +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hola\n", 5) => 5; + lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; + lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hello", "c/hello") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the source + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + // corrupt the destination + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move file after corrupt +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hola\n", 5) => 5; + lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; + lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hello", "c/hello") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the source + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + // corrupt the destination + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + // continue move + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hello", "c/hello") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # simple reentrant move file +reentrant = true +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + err = lfs_mkdir(&lfs, "a"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "b"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "c"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "d"); + assert(!err || err == LFS_ERR_EXIST); + lfs_unmount(&lfs) => 0; + + while (true) { + lfs_mount(&lfs, &cfg) => 0; + // there should never exist _2_ hello files + int count = 0; + if (lfs_stat(&lfs, "a/hello", &info) == 0) { + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6 || info.size == 0); + count += 1; + } + if (lfs_stat(&lfs, "b/hello", &info) == 0) { + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + count += 1; + } + if (lfs_stat(&lfs, "c/hello", &info) == 0) { + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + count += 1; + } + if (lfs_stat(&lfs, "d/hello", &info) == 0) { + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + count += 1; + } + assert(count <= 1); + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + if (lfs_stat(&lfs, "a/hello", &info) == 0 && info.size > 0) { + lfs_rename(&lfs, "a/hello", "b/hello") => 0; + } else if (lfs_stat(&lfs, "b/hello", &info) == 0) { + lfs_rename(&lfs, "b/hello", "c/hello") => 0; + } else if (lfs_stat(&lfs, "c/hello", &info) == 0) { + lfs_rename(&lfs, "c/hello", "d/hello") => 0; + } else if (lfs_stat(&lfs, "d/hello", &info) == 0) { + // success + break; + } else { + // create file + lfs_file_open(&lfs, &file, "a/hello", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "hola\n", 5) => 5; + lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; + lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + } + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "d") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 5+8+6); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move dir +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_mkdir(&lfs, "a/hi") => 0; + lfs_mkdir(&lfs, "a/hi/hola") => 0; + lfs_mkdir(&lfs, "a/hi/bonjour") => 0; + lfs_mkdir(&lfs, "a/hi/ohayo") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hi", "c/hi") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "a/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "b/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "c/hi") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "bonjour") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hola") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "ohayo") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "d/hi") => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move dir corrupt source +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_mkdir(&lfs, "a/hi") => 0; + lfs_mkdir(&lfs, "a/hi/hola") => 0; + lfs_mkdir(&lfs, "a/hi/bonjour") => 0; + lfs_mkdir(&lfs, "a/hi/ohayo") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hi", "c/hi") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the source + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "a/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "b/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "c/hi") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "bonjour") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hola") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "ohayo") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "d/hi") => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move dir corrupt source and dest +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_mkdir(&lfs, "a/hi") => 0; + lfs_mkdir(&lfs, "a/hi/hola") => 0; + lfs_mkdir(&lfs, "a/hi/bonjour") => 0; + lfs_mkdir(&lfs, "a/hi/ohayo") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hi", "c/hi") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the source + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + // corrupt the destination + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "a/hi") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "bonjour") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hola") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "ohayo") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "b/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "c/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "d/hi") => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move dir after corrupt +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_mkdir(&lfs, "a/hi") => 0; + lfs_mkdir(&lfs, "a/hi/hola") => 0; + lfs_mkdir(&lfs, "a/hi/bonjour") => 0; + lfs_mkdir(&lfs, "a/hi/ohayo") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hi", "c/hi") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the source + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + // corrupt the destination + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + // continue move + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hi", "c/hi") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "c") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "a/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "b/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "c/hi") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "bonjour") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hola") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "ohayo") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "d/hi") => LFS_ERR_NOENT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # simple reentrant move dir +reentrant = true +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + err = lfs_mkdir(&lfs, "a"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "b"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "c"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "d"); + assert(!err || err == LFS_ERR_EXIST); + lfs_unmount(&lfs) => 0; + + while (true) { + lfs_mount(&lfs, &cfg) => 0; + // there should never exist _2_ hi directories + int count = 0; + if (lfs_stat(&lfs, "a/hi", &info) == 0) { + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + count += 1; + } + if (lfs_stat(&lfs, "b/hi", &info) == 0) { + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + count += 1; + } + if (lfs_stat(&lfs, "c/hi", &info) == 0) { + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + count += 1; + } + if (lfs_stat(&lfs, "d/hi", &info) == 0) { + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + count += 1; + } + assert(count <= 1); + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + if (lfs_stat(&lfs, "a/hi", &info) == 0) { + lfs_rename(&lfs, "a/hi", "b/hi") => 0; + } else if (lfs_stat(&lfs, "b/hi", &info) == 0) { + lfs_rename(&lfs, "b/hi", "c/hi") => 0; + } else if (lfs_stat(&lfs, "c/hi", &info) == 0) { + lfs_rename(&lfs, "c/hi", "d/hi") => 0; + } else if (lfs_stat(&lfs, "d/hi", &info) == 0) { + break; // success + } else { + // create dir and rename for atomicity + err = lfs_mkdir(&lfs, "temp"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "temp/hola"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "temp/bonjour"); + assert(!err || err == LFS_ERR_EXIST); + err = lfs_mkdir(&lfs, "temp/ohayo"); + assert(!err || err == LFS_ERR_EXIST); + lfs_rename(&lfs, "temp", "a/hi") => 0; + } + lfs_unmount(&lfs) => 0; + } + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "a") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_dir_open(&lfs, &dir, "d") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "a/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "b/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "c/hi") => LFS_ERR_NOENT; + lfs_dir_open(&lfs, &dir, "d/hi") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "bonjour") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "hola") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "ohayo") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move state stealing +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "a") => 0; + lfs_mkdir(&lfs, "b") => 0; + lfs_mkdir(&lfs, "c") => 0; + lfs_mkdir(&lfs, "d") => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hola\n", 5) => 5; + lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; + lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "a/hello", "b/hello") => 0; + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "b/hello", "c/hello") => 0; + lfs_unmount(&lfs) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_rename(&lfs, "c/hello", "d/hello") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_remove(&lfs, "b") => 0; + lfs_remove(&lfs, "c") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "a", &info) => 0; + lfs_stat(&lfs, "b", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "c", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "d", &info) => 0; + lfs_file_open(&lfs, &file, "a/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "b/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "c/hello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 5) => 5; + memcmp(buffer, "hola\n", 5) => 0; + lfs_file_read(&lfs, &file, buffer, 8) => 8; + memcmp(buffer, "bonjour\n", 8) => 0; + lfs_file_read(&lfs, &file, buffer, 6) => 6; + memcmp(buffer, "ohayo\n", 6) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' diff --git a/tests_/test_orphan.toml b/tests_/test_orphan.toml new file mode 100644 index 00000000..fe521992 --- /dev/null +++ b/tests_/test_orphan.toml @@ -0,0 +1,56 @@ +[[case]] # orphan test +in = "lfs.c" +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "parent") => 0; + lfs_mkdir(&lfs, "parent/orphan") => 0; + lfs_mkdir(&lfs, "parent/child") => 0; + lfs_remove(&lfs, "parent/orphan") => 0; + lfs_unmount(&lfs) => 0; + + // corrupt the child's most recent commit, this should be the update + // to the linked-list entry, which should orphan the orphan. Note this + // makes a lot of assumptions about the remove operation. + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "parent/child") => 0; + lfs_block_t block = dir.m.pair[0]; + lfs_dir_close(&lfs, &dir) => 0; + lfs_unmount(&lfs) => 0; + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + int off = LFS_BLOCK_SIZE-1; + while (off >= 0 && bbuffer[off] == LFS_ERASE_VALUE) { + off -= 1; + } + memset(&bbuffer[off-3], LFS_BLOCK_SIZE, 3); + cfg.erase(&cfg, block) => 0; + cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + cfg.sync(&cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "parent/child", &info) => 0; + lfs_fs_size(&lfs) => 8; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "parent/child", &info) => 0; + lfs_fs_size(&lfs) => 8; + // this mkdir should both create a dir and deorphan, so size + // should be unchanged + lfs_mkdir(&lfs, "parent/otherchild") => 0; + lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "parent/child", &info) => 0; + lfs_stat(&lfs, "parent/otherchild", &info) => 0; + lfs_fs_size(&lfs) => 8; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; + lfs_stat(&lfs, "parent/child", &info) => 0; + lfs_stat(&lfs, "parent/otherchild", &info) => 0; + lfs_fs_size(&lfs) => 8; + lfs_unmount(&lfs) => 0; +''' diff --git a/tests_/test_seek.toml b/tests_/test_seek.toml index 0a1d90d1..586ab719 100644 --- a/tests_/test_seek.toml +++ b/tests_/test_seek.toml @@ -1,5 +1,13 @@ [[case]] # simple file seek +define = [ + {COUNT=132, SKIP=4}, + {COUNT=132, SKIP=128}, + {COUNT=200, SKIP=10}, + {COUNT=200, SKIP=100}, + {COUNT=4, SKIP=1}, + {COUNT=4, SKIP=2}, +] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -59,6 +67,8 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' + +[[case]] # simple file seek and write define = [ {COUNT=132, SKIP=4}, {COUNT=132, SKIP=128}, @@ -67,8 +77,6 @@ define = [ {COUNT=4, SKIP=1}, {COUNT=4, SKIP=2}, ] - -[[case]] # simple file seek and write code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -120,16 +128,10 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define = [ - {COUNT=132, SKIP=4}, - {COUNT=132, SKIP=128}, - {COUNT=200, SKIP=10}, - {COUNT=200, SKIP=100}, - {COUNT=4, SKIP=1}, - {COUNT=4, SKIP=2}, -] [[case]] # boundary seek and writes +define.COUNT = 132 +define.OFFSETS = '"{512, 1020, 513, 1021, 511, 1019, 1441}"' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -180,10 +182,16 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.COUNT = 132 -define.OFFSETS = '"{512, 1020, 513, 1021, 511, 1019, 1441}"' [[case]] # out of bounds seek +define = [ + {COUNT=132, SKIP=4}, + {COUNT=132, SKIP=128}, + {COUNT=200, SKIP=10}, + {COUNT=200, SKIP=100}, + {COUNT=4, SKIP=2}, + {COUNT=4, SKIP=3}, +] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -229,16 +237,9 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define = [ - {COUNT=132, SKIP=4}, - {COUNT=132, SKIP=128}, - {COUNT=200, SKIP=10}, - {COUNT=200, SKIP=100}, - {COUNT=4, SKIP=2}, - {COUNT=4, SKIP=3}, -] [[case]] # inline write and seek +define.SIZE = [2, 4, 128, 132] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -303,9 +304,11 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.SIZE = [2, 4, 128, 132] [[case]] # file seek and write with power-loss +# must be power-of-2 for quadratic probing to be exhaustive +define.COUNT = [4, 64, 128] +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -375,6 +378,3 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -# must be power-of-2 for quadratic probing to be exhaustive -define.COUNT = [4, 64, 128] -reentrant = true diff --git a/tests_/test_truncate.toml b/tests_/test_truncate.toml index fde2da51..44169862 100644 --- a/tests_/test_truncate.toml +++ b/tests_/test_truncate.toml @@ -1,4 +1,6 @@ [[case]] # simple truncate +define.MEDIUMSIZE = [32, 2048] +define.LARGESIZE = 8192 code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -39,10 +41,10 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.MEDIUMSIZE = [32, 2048] -define.LARGESIZE = 8192 [[case]] # truncate and read +define.MEDIUMSIZE = [32, 2048] +define.LARGESIZE = 8192 code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -90,8 +92,6 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.MEDIUMSIZE = [32, 2048] -define.LARGESIZE = 8192 [[case]] # write, truncate, and read code = ''' @@ -146,6 +146,8 @@ code = ''' ''' [[case]] # truncate and write +define.MEDIUMSIZE = [32, 2048] +define.LARGESIZE = 8192 code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -193,10 +195,12 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' -define.MEDIUMSIZE = [32, 2048] -define.LARGESIZE = 8192 [[case]] # truncate write under powerloss +define.SMALLSIZE = [4, 512] +define.MEDIUMSIZE = [32, 1024] +define.LARGESIZE = 2048 +reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -257,12 +261,12 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -define.SMALLSIZE = [4, 512] -define.MEDIUMSIZE = [32, 1024] -define.LARGESIZE = 2048 -reentrant = true [[case]] # more aggressive general truncation tests +define.CONFIG = 'range(6)' +define.SMALLSIZE = 32 +define.MEDIUMSIZE = 2048 +define.LARGESIZE = 8192 code = ''' #define COUNT 5 const struct { @@ -388,8 +392,3 @@ code = ''' lfs_unmount(&lfs) => 0; ''' -define.CONFIG = 'range(6)' -define.SMALLSIZE = 32 -define.MEDIUMSIZE = 2048 -define.LARGESIZE = 8192 - From ecc2857c0e900d1ae1f49de507fb9851345e6ca7 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 14 Jan 2020 12:04:20 -0600 Subject: [PATCH 08/41] Migrated bad-block tests Even with adding better reentrance testing, the bad-block tests are still very useful at isolating the block eviction logic. This also required rewriting a bit of the internal testing wirework to allow custom block devices which opens up quite a bit more straegies for testing. --- scripts/test_.py | 124 +++++++++--- tests_/test_badblocks.toml | 375 +++++++++++++++++++++++++++++++++++++ 2 files changed, 470 insertions(+), 29 deletions(-) create mode 100644 tests_/test_badblocks.toml diff --git a/scripts/test_.py b/scripts/test_.py index 760f64e7..2b785109 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -19,6 +19,7 @@ # x config chaining correct # - why can't gdb see my defines? # - say no to internal? +# - buffering stdout issues? import toml import glob @@ -55,8 +56,79 @@ #include "filebd/lfs_filebd.h" #include "rambd/lfs_rambd.h" #include + +extern const char *lfs_testbd_disk; +typedef union { + lfs_filebd_t filebd; + lfs_rambd_t rambd; +} lfs_testbd_t; +struct lfs_testbd_config { + struct lfs_filebd_config filecfg; + struct lfs_rambd_config ramcfg; +}; + +__attribute__((unused)) +static int lfs_testbd_createcfg(const struct lfs_config *cfg, + const struct lfs_testbd_config *bdcfg) { + if (lfs_testbd_disk) { + return lfs_filebd_createcfg(cfg, lfs_testbd_disk, &bdcfg->filecfg); + } else { + return lfs_rambd_createcfg(cfg, &bdcfg->ramcfg); + } +} + +__attribute__((unused)) +static void lfs_testbd_destroy(const struct lfs_config *cfg) { + if (lfs_testbd_disk) { + lfs_filebd_destroy(cfg); + } else { + lfs_rambd_destroy(cfg); + } +} + +__attribute__((unused)) +static int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size) { + if (lfs_testbd_disk) { + return lfs_filebd_read(cfg, block, off, buffer, size); + } else { + return lfs_rambd_read(cfg, block, off, buffer, size); + } +} + +__attribute__((unused)) +static int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size) { + if (lfs_testbd_disk) { + return lfs_filebd_prog(cfg, block, off, buffer, size); + } else { + return lfs_rambd_prog(cfg, block, off, buffer, size); + } +} + +__attribute__((unused)) +static int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { + if (lfs_testbd_disk) { + return lfs_filebd_erase(cfg, block); + } else { + return lfs_rambd_erase(cfg, block); + } +} + +__attribute__((unused)) +static int lfs_testbd_sync(const struct lfs_config *cfg) { + if (lfs_testbd_disk) { + return lfs_filebd_sync(cfg); + } else { + return lfs_rambd_sync(cfg); + } +} """ DEFINES = { + "LFS_BD_READ": "lfs_testbd_read", + "LFS_BD_PROG": "lfs_testbd_prog", + "LFS_BD_ERASE": "lfs_testbd_erase", + "LFS_BD_SYNC": "lfs_testbd_sync", "LFS_READ_SIZE": 16, "LFS_PROG_SIZE": "LFS_READ_SIZE", "LFS_BLOCK_SIZE": 512, @@ -68,11 +140,8 @@ } PROLOGUE = """ // prologue - extern const char *LFS_DISK; - __attribute__((unused)) lfs_t lfs; - __attribute__((unused)) lfs_filebd_t filebd; - __attribute__((unused)) lfs_rambd_t rambd; + __attribute__((unused)) lfs_testbd_t bd; __attribute__((unused)) lfs_file_t file; __attribute__((unused)) lfs_dir_t dir; __attribute__((unused)) struct lfs_info info; @@ -82,12 +151,11 @@ __attribute__((unused)) int err; __attribute__((unused)) const struct lfs_config cfg = { - .context = LFS_DISK ? (void*)&filebd : (void*)&rambd, - .read = LFS_DISK ? &lfs_filebd_read : &lfs_rambd_read, - .prog = LFS_DISK ? &lfs_filebd_prog : &lfs_rambd_prog, - .erase = LFS_DISK ? &lfs_filebd_erase : &lfs_rambd_erase, - .sync = LFS_DISK ? &lfs_filebd_sync : &lfs_rambd_sync, - + .context = &bd, + .read = LFS_BD_READ, + .prog = LFS_BD_PROG, + .erase = LFS_BD_ERASE, + .sync = LFS_BD_SYNC, .read_size = LFS_READ_SIZE, .prog_size = LFS_PROG_SIZE, .block_size = LFS_BLOCK_SIZE, @@ -97,26 +165,16 @@ .lookahead_size = LFS_LOOKAHEAD_SIZE, }; - __attribute__((unused)) const struct lfs_filebd_config filecfg = { - .erase_value = LFS_ERASE_VALUE, - }; - __attribute__((unused)) const struct lfs_rambd_config ramcfg = { - .erase_value = LFS_ERASE_VALUE, + __attribute__((unused)) const struct lfs_testbd_config bdcfg = { + .filecfg.erase_value = LFS_ERASE_VALUE, + .ramcfg.erase_value = LFS_ERASE_VALUE, }; - if (LFS_DISK) { - lfs_filebd_createcfg(&cfg, LFS_DISK, &filecfg); - } else { - lfs_rambd_createcfg(&cfg, &ramcfg); - } + lfs_testbd_createcfg(&cfg, &bdcfg) => 0; """ EPILOGUE = """ // epilogue - if (LFS_DISK) { - lfs_filebd_destroy(&cfg); - } else { - lfs_rambd_destroy(&cfg); - } + lfs_testbd_destroy(&cfg); """ PASS = '\033[32m✓\033[0m' FAIL = '\033[31m✗\033[0m' @@ -363,15 +421,20 @@ def __init__(self, path, filter=None, TestCase=TestCase, **args): if re.match(r'code\s*=\s*(\'\'\'|""")', line): code_linenos.append(i+2) + code_linenos.reverse() + # grab global config self.defines = config.get('define', {}) + self.code = config.get('code', None) + if self.code is not None: + self.code_lineno = code_linenos.pop() # create initial test cases self.cases = [] for i, (case, lineno) in enumerate(zip(config['case'], linenos)): # code lineno? if 'code' in case: - case['code_lineno'] = code_linenos.pop(0) + case['code_lineno'] = code_linenos.pop() # give our case's config a copy of our "global" config for k, v in config.items(): if k not in case: @@ -452,8 +515,11 @@ def build(self, **args): # build test files tf = open(self.path + '.test.c.t', 'w') tf.write(GLOBALS) - tfs = {None: tf} + if self.code is not None: + tf.write('#line %d "%s"\n' % (self.code_lineno, self.path)) + tf.write(self.code) + tfs = {None: tf} for case in self.cases: if case.in_ not in tfs: tfs[case.in_] = open(self.path+'.'+ @@ -469,11 +535,11 @@ def build(self, **args): case.build(tfs[case.in_], **args) tf.write('\n') - tf.write('const char *LFS_DISK = NULL;\n') + tf.write('const char *lfs_testbd_disk;\n') tf.write('int main(int argc, char **argv) {\n') tf.write(4*' '+'int case_ = (argc >= 2) ? atoi(argv[1]) : 0;\n') tf.write(4*' '+'int perm = (argc >= 3) ? atoi(argv[2]) : 0;\n') - tf.write(4*' '+'LFS_DISK = (argc >= 4) ? argv[3] : NULL;\n') + tf.write(4*' '+'lfs_testbd_disk = (argc >= 4) ? argv[3] : NULL;\n') for perm in self.perms: # test declaration tf.write(4*' '+'extern void test_case%d(%s);\n' % ( diff --git a/tests_/test_badblocks.toml b/tests_/test_badblocks.toml new file mode 100644 index 00000000..7599647c --- /dev/null +++ b/tests_/test_badblocks.toml @@ -0,0 +1,375 @@ +# special bad-block block device hook for simulating blocks +# that are no longer writable +code = ''' +lfs_block_t *bbbd_badblocks; +size_t bbbd_badblocks_count; + +int bbbd_read(const struct lfs_config *c, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size) { + for (size_t i = 0; i < bbbd_badblocks_count; i++) { + if (bbbd_badblocks[i] == block) { + return LFS_ERR_CORRUPT; + } + } + + return lfs_testbd_read(c, block, off, buffer, size); +} + +int bbbd_prog(const struct lfs_config *c, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size) { + for (size_t i = 0; i < bbbd_badblocks_count; i++) { + if (bbbd_badblocks[i] == block) { + return LFS_ERR_CORRUPT; + } + } + + return lfs_testbd_prog(c, block, off, buffer, size); +} + +int bbbd_erase(const struct lfs_config *c, lfs_block_t block) { + for (size_t i = 0; i < bbbd_badblocks_count; i++) { + if (bbbd_badblocks[i] == block) { + return LFS_ERR_CORRUPT; + } + } + + return lfs_testbd_erase(c, block); +} +''' + +[[case]] # single bad blocks +define.LFS_BD_PROG = 'bbbd_prog' +define.NAMEMULT = 64 +define.FILEMULT = 1 +code = ''' + for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { + bbbd_badblocks = &(lfs_block_t){badblock}; + bbbd_badblocks_count = 1; + + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_mkdir(&lfs, (char*)buffer) => 0; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_stat(&lfs, (char*)buffer, &info) => 0; + info.type => LFS_TYPE_DIR; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(buffer, rbuffer, size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + } +''' + +[[case]] # single persistent blocks (can't erase) +define.LFS_BD_ERASE = 'bbbd_erase' +define.NAMEMULT = 64 +define.FILEMULT = 1 +code = ''' + for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { + bbbd_badblocks = &(lfs_block_t){badblock}; + bbbd_badblocks_count = 1; + + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_mkdir(&lfs, (char*)buffer) => 0; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_stat(&lfs, (char*)buffer, &info) => 0; + info.type => LFS_TYPE_DIR; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(buffer, rbuffer, size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + } +''' + +[[case]] # single unreadable blocks (can't read) +define.LFS_BD_READ = 'bbbd_read' +define.NAMEMULT = 64 +define.FILEMULT = 1 +code = ''' + for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { + bbbd_badblocks = &(lfs_block_t){badblock}; + bbbd_badblocks_count = 1; + + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_mkdir(&lfs, (char*)buffer) => 0; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_stat(&lfs, (char*)buffer, &info) => 0; + info.type => LFS_TYPE_DIR; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(buffer, rbuffer, size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + } +''' + +[[case]] # region corruption (causes cascading failures) +define.LFS_BD_PROG = '"BADTYPE == 0 ? bbbd_prog : lfs_testbd_prog "' +define.LFS_BD_ERASE = '"BADTYPE == 1 ? bbbd_erase : lfs_testbd_erase"' +define.LFS_BD_READ = '"BADTYPE == 2 ? bbbd_read : lfs_testbd_read "' +define.BADTYPE = 'range(3)' +define.NAMEMULT = 64 +define.FILEMULT = 1 +code = ''' + bbbd_badblocks_count = LFS_BLOCK_COUNT/2; + bbbd_badblocks = malloc(bbbd_badblocks_count*sizeof(lfs_block_t)); + for (size_t i = 0; i < bbbd_badblocks_count; i++) { + bbbd_badblocks[i] = i+2; + } + + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_mkdir(&lfs, (char*)buffer) => 0; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_stat(&lfs, (char*)buffer, &info) => 0; + info.type => LFS_TYPE_DIR; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(buffer, rbuffer, size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + free(bbbd_badblocks); +''' + +[[case]] # alternating corruption (causes cascading failures) +define.LFS_BD_PROG = '"BADTYPE == 0 ? bbbd_prog : lfs_testbd_prog "' +define.LFS_BD_ERASE = '"BADTYPE == 1 ? bbbd_erase : lfs_testbd_erase"' +define.LFS_BD_READ = '"BADTYPE == 2 ? bbbd_read : lfs_testbd_read "' +define.BADTYPE = 'range(3)' +define.NAMEMULT = 64 +define.FILEMULT = 1 +code = ''' + bbbd_badblocks_count = LFS_BLOCK_COUNT/2; + bbbd_badblocks = malloc(bbbd_badblocks_count*sizeof(lfs_block_t)); + for (size_t i = 0; i < bbbd_badblocks_count; i++) { + bbbd_badblocks[i] = (2*i) + 2; + } + + lfs_format(&lfs, &cfg) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_mkdir(&lfs, (char*)buffer) => 0; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, + LFS_O_WRONLY | LFS_O_CREAT) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int i = 1; i < 10; i++) { + for (int j = 0; j < NAMEMULT; j++) { + buffer[j] = '0'+i; + } + buffer[NAMEMULT] = '\0'; + lfs_stat(&lfs, (char*)buffer, &info) => 0; + info.type => LFS_TYPE_DIR; + + buffer[NAMEMULT] = '/'; + for (int j = 0; j < NAMEMULT; j++) { + buffer[j+NAMEMULT+1] = '0'+i; + } + buffer[2*NAMEMULT+1] = '\0'; + lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; + + size = NAMEMULT; + for (int j = 0; j < i*FILEMULT; j++) { + uint8_t rbuffer[1024]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + memcmp(buffer, rbuffer, size) => 0; + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + free(bbbd_badblocks); +''' From fb65057a3c2c2b18841086c25ee172f5441bd9ca Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Thu, 16 Jan 2020 06:30:40 -0600 Subject: [PATCH 09/41] Restructured block devices again for better test exploitation Also finished migrating tests with test_relocations and test_exhaustion. The issue I was running into when migrating these tests was a lack of flexibility with what you could do with the block devices. It was possible to hack in some hooks for things like bad blocks and power loss, but it wasn't clean or easily extendable. The solution here was to just put all of these test extensions into a third block device, testbd, that uses the other two example block devices internally. testbd has several useful features for testing. Note this makes it a pretty terrible block device _example_ since these hooks look more complicated than a block device needs to be. - testbd can simulate different erase values, supporting 1s, 0s, other byte patterns, or no erases at all (which can cause surprising bugs). This actually depends on the simulated erase values in ramdb and filebd. I did try to move this out of rambd/filebd, but it's not possible to simulate erases in testbd without buffering entire blocks and creating an excessive amount of extra write operations. - testbd also helps simulate power-loss by containing a "power cycles" counter that is decremented every write operation until it calls exit. This is notably faster than the previous gdb approach, which is valuable since the reentrant tests tend to take a while to resolve. - testbd also tracks wear, which can be manually set and read. This is very useful for testing things like bad block handling, wear leveling, or even changing the effective size of the block device at runtime. --- Makefile | 2 +- filebd/lfs_filebd.c | 18 +- filebd/lfs_filebd.h | 9 +- lfs.c | 4 +- rambd/lfs_rambd.c | 11 +- rambd/lfs_rambd.h | 4 +- scripts/explode_asserts.py | 6 +- scripts/test_.py | 227 +++++++++-------------- testbd/lfs_testbd.c | 289 +++++++++++++++++++++++++++++ testbd/lfs_testbd.h | 130 +++++++++++++ tests_/test_alloc.toml | 2 + tests_/test_attrs.toml | 3 +- tests_/test_badblocks.toml | 112 +++++------- tests_/test_exhaustion.toml | 341 +++++++++++++++++++++++++++++++++++ tests_/test_format.toml | 2 - tests_/test_relocations.toml | 143 +++++++++++++++ 16 files changed, 1066 insertions(+), 237 deletions(-) create mode 100644 testbd/lfs_testbd.c create mode 100644 testbd/lfs_testbd.h create mode 100644 tests_/test_exhaustion.toml create mode 100644 tests_/test_relocations.toml diff --git a/Makefile b/Makefile index 17d3ab38..67e1c9e7 100644 --- a/Makefile +++ b/Makefile @@ -7,7 +7,7 @@ CC ?= gcc AR ?= ar SIZE ?= size -SRC += $(wildcard *.c rambd/*.c filebd/*.c) +SRC += $(wildcard *.c rambd/*.c filebd/*.c testbd/*.c) OBJ := $(SRC:.c=.o) DEP := $(SRC:.c=.d) ASM := $(SRC:.c=.s) diff --git a/filebd/lfs_filebd.c b/filebd/lfs_filebd.c index d7fd89f5..a897a02d 100644 --- a/filebd/lfs_filebd.c +++ b/filebd/lfs_filebd.c @@ -11,7 +11,7 @@ #include int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, - const struct lfs_filebd_config *filecfg) { + const struct lfs_filebd_config *bdcfg) { LFS_TRACE("lfs_filebd_createcfg(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " @@ -22,9 +22,9 @@ int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, - path, (void*)filecfg, filecfg->erase_value); + path, (void*)bdcfg, bdcfg->erase_value); lfs_filebd_t *bd = cfg->context; - bd->cfg = filecfg; + bd->cfg = bdcfg; // open file bd->fd = open(path, O_RDWR | O_CREAT, 0666); @@ -55,11 +55,17 @@ int lfs_filebd_create(const struct lfs_config *cfg, const char *path) { return err; } -void lfs_filebd_destroy(const struct lfs_config *cfg) { +int lfs_filebd_destroy(const struct lfs_config *cfg) { LFS_TRACE("lfs_filebd_destroy(%p)", (void*)cfg); lfs_filebd_t *bd = cfg->context; - close(bd->fd); - LFS_TRACE("lfs_filebd_destroy -> %s", "void"); + int err = close(bd->fd); + if (err < 0) { + err = -errno; + LFS_TRACE("lfs_filebd_destroy -> %d", err); + return err; + } + LFS_TRACE("lfs_filebd_destroy -> %d", 0); + return 0; } int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block, diff --git a/filebd/lfs_filebd.h b/filebd/lfs_filebd.h index ae288582..271e873d 100644 --- a/filebd/lfs_filebd.h +++ b/filebd/lfs_filebd.h @@ -17,8 +17,9 @@ extern "C" // filebd config (optional) struct lfs_filebd_config { - // 8-bit erase value to simulate erasing with. -1 indicates no erase - // occurs, which is still a valid block device + // 8-bit erase value to use for simulating erases. -1 does not simulate + // erases, which can speed up testing by avoiding all the extra block-device + // operations to store the erase value. int32_t erase_value; }; @@ -32,10 +33,10 @@ typedef struct lfs_filebd { // Create a file block device using the geometry in lfs_config int lfs_filebd_create(const struct lfs_config *cfg, const char *path); int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, - const struct lfs_filebd_config *ramcfg); + const struct lfs_filebd_config *bdcfg); // Clean up memory associated with block device -void lfs_filebd_destroy(const struct lfs_config *cfg); +int lfs_filebd_destroy(const struct lfs_config *cfg); // Read a block int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block, diff --git a/lfs.c b/lfs.c index 4553a6ed..2758c29c 100644 --- a/lfs.c +++ b/lfs.c @@ -1651,9 +1651,11 @@ static int lfs_dir_compact(lfs_t *lfs, // relocate half of pair int err = lfs_alloc(lfs, &dir->pair[1]); - if (err && (err != LFS_ERR_NOSPC && !exhausted)) { + if (err && (err != LFS_ERR_NOSPC || !exhausted)) { return err; } + + exhausted = false; continue; } diff --git a/rambd/lfs_rambd.c b/rambd/lfs_rambd.c index 5bf8cc13..ce065563 100644 --- a/rambd/lfs_rambd.c +++ b/rambd/lfs_rambd.c @@ -7,7 +7,7 @@ #include "rambd/lfs_rambd.h" int lfs_rambd_createcfg(const struct lfs_config *cfg, - const struct lfs_rambd_config *ramcfg) { + const struct lfs_rambd_config *bdcfg) { LFS_TRACE("lfs_rambd_createcfg(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " @@ -17,9 +17,9 @@ int lfs_rambd_createcfg(const struct lfs_config *cfg, (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, - (void*)ramcfg, ramcfg->erase_value, ramcfg->buffer); + (void*)bdcfg, bdcfg->erase_value, bdcfg->buffer); lfs_rambd_t *bd = cfg->context; - bd->cfg = ramcfg; + bd->cfg = bdcfg; // allocate buffer? if (bd->cfg->buffer) { @@ -57,14 +57,15 @@ int lfs_rambd_create(const struct lfs_config *cfg) { return err; } -void lfs_rambd_destroy(const struct lfs_config *cfg) { +int lfs_rambd_destroy(const struct lfs_config *cfg) { LFS_TRACE("lfs_rambd_destroy(%p)", (void*)cfg); // clean up memory lfs_rambd_t *bd = cfg->context; if (!bd->cfg->buffer) { lfs_free(bd->buffer); } - LFS_TRACE("lfs_rambd_destroy -> %s", "void"); + LFS_TRACE("lfs_rambd_destroy -> %d", 0); + return 0; } int lfs_rambd_read(const struct lfs_config *cfg, lfs_block_t block, diff --git a/rambd/lfs_rambd.h b/rambd/lfs_rambd.h index a4b95151..139ddd67 100644 --- a/rambd/lfs_rambd.h +++ b/rambd/lfs_rambd.h @@ -35,10 +35,10 @@ typedef struct lfs_rambd { // Create a RAM block device using the geometry in lfs_config int lfs_rambd_create(const struct lfs_config *cfg); int lfs_rambd_createcfg(const struct lfs_config *cfg, - const struct lfs_rambd_config *ramcfg); + const struct lfs_rambd_config *bdcfg); // Clean up memory associated with block device -void lfs_rambd_destroy(const struct lfs_config *cfg); +int lfs_rambd_destroy(const struct lfs_config *cfg); // Read a block int lfs_rambd_read(const struct lfs_config *cfg, lfs_block_t block, diff --git a/scripts/explode_asserts.py b/scripts/explode_asserts.py index 7c24c633..ff3f260a 100755 --- a/scripts/explode_asserts.py +++ b/scripts/explode_asserts.py @@ -146,7 +146,7 @@ def pnested(): pexpr = ( # shortcut for a bit better performance - p.regex('[^%s/#\'"();{}=><,&|-]+' % ASSERT_CHARS) | + p.regex('[^%s/#\'"():;{}=><,&|-]+' % ASSERT_CHARS) | pws | passert | pstring | @@ -157,7 +157,7 @@ def pnested(): @p.generate def pstmt(): ws = yield pws.many() - lh = yield pexpr.until(p.string('=>') | p.regex('[;{}]')) + lh = yield pexpr.until(p.string('=>') | p.regex('[:;{}]')) op = yield p.string('=>').optional() if op == '=>': rh = yield pstmt @@ -168,7 +168,7 @@ def pstmt(): @p.generate def pstmts(): a = yield pstmt - b = yield (p.regex('[;{}]') + pstmt).many() + b = yield (p.regex('[:;{}]') + pstmt).many() return [a] + b def main(args): diff --git a/scripts/test_.py b/scripts/test_.py index 2b785109..c4e44a12 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -19,7 +19,7 @@ # x config chaining correct # - why can't gdb see my defines? # - say no to internal? -# - buffering stdout issues? +# x buffering stdout issues? import toml import glob @@ -33,6 +33,9 @@ import sys import copy import shlex +import pty +import errno +import signal TESTDIR = 'tests_' RULES = """ @@ -45,98 +48,31 @@ -include tests_/*.d .SECONDARY: -%.test: override CFLAGS += -fdiagnostics-color=always -%.test: override CFLAGS += -ggdb +%.test: override CFLAGS += -gdwarf-2 +%.test: override CFLAGS += -ggdb3 +%.test: override CFLAGS += -g3 %.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.$f) $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ """ GLOBALS = """ //////////////// AUTOGENERATED TEST //////////////// #include "lfs.h" -#include "filebd/lfs_filebd.h" -#include "rambd/lfs_rambd.h" +#include "testbd/lfs_testbd.h" #include - -extern const char *lfs_testbd_disk; -typedef union { - lfs_filebd_t filebd; - lfs_rambd_t rambd; -} lfs_testbd_t; -struct lfs_testbd_config { - struct lfs_filebd_config filecfg; - struct lfs_rambd_config ramcfg; -}; - -__attribute__((unused)) -static int lfs_testbd_createcfg(const struct lfs_config *cfg, - const struct lfs_testbd_config *bdcfg) { - if (lfs_testbd_disk) { - return lfs_filebd_createcfg(cfg, lfs_testbd_disk, &bdcfg->filecfg); - } else { - return lfs_rambd_createcfg(cfg, &bdcfg->ramcfg); - } -} - -__attribute__((unused)) -static void lfs_testbd_destroy(const struct lfs_config *cfg) { - if (lfs_testbd_disk) { - lfs_filebd_destroy(cfg); - } else { - lfs_rambd_destroy(cfg); - } -} - -__attribute__((unused)) -static int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, - lfs_off_t off, void *buffer, lfs_size_t size) { - if (lfs_testbd_disk) { - return lfs_filebd_read(cfg, block, off, buffer, size); - } else { - return lfs_rambd_read(cfg, block, off, buffer, size); - } -} - -__attribute__((unused)) -static int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, - lfs_off_t off, const void *buffer, lfs_size_t size) { - if (lfs_testbd_disk) { - return lfs_filebd_prog(cfg, block, off, buffer, size); - } else { - return lfs_rambd_prog(cfg, block, off, buffer, size); - } -} - -__attribute__((unused)) -static int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { - if (lfs_testbd_disk) { - return lfs_filebd_erase(cfg, block); - } else { - return lfs_rambd_erase(cfg, block); - } -} - -__attribute__((unused)) -static int lfs_testbd_sync(const struct lfs_config *cfg) { - if (lfs_testbd_disk) { - return lfs_filebd_sync(cfg); - } else { - return lfs_rambd_sync(cfg); - } -} +extern const char *lfs_testbd_path; +extern uint32_t lfs_testbd_cycles; """ DEFINES = { - "LFS_BD_READ": "lfs_testbd_read", - "LFS_BD_PROG": "lfs_testbd_prog", - "LFS_BD_ERASE": "lfs_testbd_erase", - "LFS_BD_SYNC": "lfs_testbd_sync", - "LFS_READ_SIZE": 16, - "LFS_PROG_SIZE": "LFS_READ_SIZE", - "LFS_BLOCK_SIZE": 512, - "LFS_BLOCK_COUNT": 1024, - "LFS_BLOCK_CYCLES": -1, - "LFS_CACHE_SIZE": "(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)", - "LFS_LOOKAHEAD_SIZE": 16, - "LFS_ERASE_VALUE": 0xff, + 'LFS_READ_SIZE': 16, + 'LFS_PROG_SIZE': 'LFS_READ_SIZE', + 'LFS_BLOCK_SIZE': 512, + 'LFS_BLOCK_COUNT': 1024, + 'LFS_BLOCK_CYCLES': -1, + 'LFS_CACHE_SIZE': '(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)', + 'LFS_LOOKAHEAD_SIZE': 16, + 'LFS_ERASE_VALUE': 0xff, + 'LFS_ERASE_CYCLES': 0, + 'LFS_BADBLOCK_BEHAVIOR': 'LFS_TESTBD_BADBLOCK_NOPROG', } PROLOGUE = """ // prologue @@ -152,10 +88,10 @@ __attribute__((unused)) const struct lfs_config cfg = { .context = &bd, - .read = LFS_BD_READ, - .prog = LFS_BD_PROG, - .erase = LFS_BD_ERASE, - .sync = LFS_BD_SYNC, + .read = lfs_testbd_read, + .prog = lfs_testbd_prog, + .erase = lfs_testbd_erase, + .sync = lfs_testbd_sync, .read_size = LFS_READ_SIZE, .prog_size = LFS_PROG_SIZE, .block_size = LFS_BLOCK_SIZE, @@ -166,15 +102,17 @@ }; __attribute__((unused)) const struct lfs_testbd_config bdcfg = { - .filecfg.erase_value = LFS_ERASE_VALUE, - .ramcfg.erase_value = LFS_ERASE_VALUE, + .erase_value = LFS_ERASE_VALUE, + .erase_cycles = LFS_ERASE_CYCLES, + .badblock_behavior = LFS_BADBLOCK_BEHAVIOR, + .power_cycles = lfs_testbd_cycles, }; - lfs_testbd_createcfg(&cfg, &bdcfg) => 0; + lfs_testbd_createcfg(&cfg, lfs_testbd_path, &bdcfg) => 0; """ EPILOGUE = """ // epilogue - lfs_testbd_destroy(&cfg); + lfs_testbd_destroy(&cfg) => 0; """ PASS = '\033[32m✓\033[0m' FAIL = '\033[31m✗\033[0m' @@ -224,15 +162,15 @@ def permute(self, defines, permno=None, **_): def build(self, f, **_): # prologue - f.write('void test_case%d(%s) {\n' % (self.caseno, ','.join( + for k, v in sorted(self.defines.items()): + if k not in self.suite.defines: + f.write('#define %s %s\n' % (k, v)) + + f.write('void test_case%d(%s) {' % (self.caseno, ','.join( '\n'+8*' '+'__attribute__((unused)) intmax_t %s' % k for k in sorted(self.perms[0].defines) if k not in self.defines))) - for k, v in sorted(self.defines.items()): - if k not in self.suite.defines: - f.write(4*' '+'#define %s %s\n' % (k, v)) - f.write(PROLOGUE) f.write('\n') f.write(4*' '+'// test case %d\n' % self.caseno) @@ -243,13 +181,11 @@ def build(self, f, **_): # epilogue f.write(EPILOGUE) - f.write('\n') + f.write('}\n') for k, v in sorted(self.defines.items()): if k not in self.suite.defines: - f.write(4*' '+'#undef %s\n' % k) - - f.write('}\n') + f.write('#undef %s\n' % k) def shouldtest(self, **args): if (self.filter is not None and @@ -265,7 +201,8 @@ def shouldtest(self, **args): else: return True - def test(self, exec=[], persist=False, gdb=False, failure=None, **args): + def test(self, exec=[], persist=False, cycles=None, + gdb=False, failure=None, **args): # build command cmd = exec + ['./%s.test' % self.suite.path, repr(self.caseno), repr(self.permno)] @@ -280,6 +217,10 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): cmd.append(self.suite.path + '.disk') + # simulate power-loss after n cycles? + if cycles: + cmd.append(str(cycles)) + # failed? drop into debugger? if gdb and failure: ncmd = ['gdb'] @@ -295,19 +236,25 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): if args.get('verbose', False): print(' '.join(shlex.quote(c) for c in ncmd)) + signal.signal(signal.SIGINT, signal.SIG_IGN) sys.exit(sp.call(ncmd)) # run test case! - stdout = [] - assert_ = None + mpty, spty = pty.openpty() if args.get('verbose', False): print(' '.join(shlex.quote(c) for c in cmd)) - proc = sp.Popen(cmd, - universal_newlines=True, - bufsize=1, - stdout=sp.PIPE, - stderr=sp.STDOUT) - for line in iter(proc.stdout.readline, ''): + proc = sp.Popen(cmd, stdout=spty, stderr=spty) + os.close(spty) + mpty = os.fdopen(mpty, 'r', 1) + stdout = [] + assert_ = None + while True: + try: + line = mpty.readline() + except OSError as e: + if e.errno == errno.EIO: + break + raise stdout.append(line) if args.get('verbose', False): sys.stdout.write(line) @@ -361,36 +308,23 @@ def shouldtest(self, **args): return self.reentrant and super().shouldtest(**args) def test(self, exec=[], persist=False, gdb=False, failure=None, **args): - # clear disk first? - if persist != 'noerase': - try: - os.remove(self.suite.path + '.disk') - except FileNotFoundError: - pass - for cycles in it.count(1): + # clear disk first? + if cycles == 1 and persist != 'noerase': + persist = 'erase' + else: + persist = 'noerase' + # exact cycle we should drop into debugger? if gdb and failure and failure.cycleno == cycles: - return super().test(exec=exec, persist='noerase', - gdb=gdb, failure=failure, **args) + return super().test(gdb=gdb, + persist=persist, failure=failure, **args) # run tests, but kill the program after prog/erase has # been hit n cycles. We exit with a special return code if the # program has not finished, since this isn't a test failure. - nexec = exec + [ - 'gdb', '-batch-silent', - '-ex', 'handle all nostop', - '-ex', 'b lfs_filebd_prog', - '-ex', 'b lfs_filebd_erase', - '-ex', 'r', - ] + cycles*['-ex', 'c'] + [ - '-ex', 'q ' - '!$_isvoid($_exitsignal) ? $_exitsignal : ' - '!$_isvoid($_exitcode) ? $_exitcode : ' - '33', - '--args'] try: - return super().test(exec=nexec, persist='noerase', **args) + return super().test(persist=persist, cycles=cycles, **args) except TestFailure as nfailure: if nfailure.returncode == 33: continue @@ -535,11 +469,13 @@ def build(self, **args): case.build(tfs[case.in_], **args) tf.write('\n') - tf.write('const char *lfs_testbd_disk;\n') + tf.write('const char *lfs_testbd_path;\n') + tf.write('uint32_t lfs_testbd_cycles;\n') tf.write('int main(int argc, char **argv) {\n') - tf.write(4*' '+'int case_ = (argc >= 2) ? atoi(argv[1]) : 0;\n') - tf.write(4*' '+'int perm = (argc >= 3) ? atoi(argv[2]) : 0;\n') - tf.write(4*' '+'lfs_testbd_disk = (argc >= 4) ? argv[3] : NULL;\n') + tf.write(4*' '+'int case_ = (argc > 1) ? atoi(argv[1]) : 0;\n') + tf.write(4*' '+'int perm = (argc > 2) ? atoi(argv[2]) : 0;\n') + tf.write(4*' '+'lfs_testbd_path = (argc > 3) ? argv[3] : NULL;\n') + tf.write(4*' '+'lfs_testbd_cycles = (argc > 4) ? atoi(argv[4]) : 0;\n') for perm in self.perms: # test declaration tf.write(4*' '+'extern void test_case%d(%s);\n' % ( @@ -671,15 +607,20 @@ def main(**args): cmd = (['make', '-f', 'Makefile'] + list(it.chain.from_iterable(['-f', m] for m in makefiles)) + [target for target in targets]) - stdout = [] + mpty, spty = pty.openpty() if args.get('verbose', False): print(' '.join(shlex.quote(c) for c in cmd)) - proc = sp.Popen(cmd, - universal_newlines=True, - bufsize=1, - stdout=sp.PIPE, - stderr=sp.STDOUT) - for line in iter(proc.stdout.readline, ''): + proc = sp.Popen(cmd, stdout=spty, stderr=spty) + os.close(spty) + mpty = os.fdopen(mpty, 'r', 1) + stdout = [] + while True: + try: + line = mpty.readline() + except OSError as e: + if e.errno == errno.EIO: + break + raise stdout.append(line) if args.get('verbose', False): sys.stdout.write(line) diff --git a/testbd/lfs_testbd.c b/testbd/lfs_testbd.c new file mode 100644 index 00000000..f13140ed --- /dev/null +++ b/testbd/lfs_testbd.c @@ -0,0 +1,289 @@ +/* + * Testing block device, wraps filebd and rambd while providing a bunch + * of hooks for testing littlefs in various conditions. + * + * Copyright (c) 2017, Arm Limited. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + */ +#include "testbd/lfs_testbd.h" + +#include + + +int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, + const struct lfs_testbd_config *bdcfg) { + LFS_TRACE("lfs_testbd_createcfg(%p {.context=%p, " + ".read=%p, .prog=%p, .erase=%p, .sync=%p, " + ".read_size=%"PRIu32", .prog_size=%"PRIu32", " + ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " + "\"%s\", " + "%p {.erase_value=%"PRId32", .erase_cycles=%"PRIu32", " + ".badblock_behavior=%"PRIu8", .power_cycles=%"PRIu32", " + ".buffer=%p, .wear_buffer=%p})", + (void*)cfg, cfg->context, + (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, + (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, + cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, + path, (void*)bdcfg, bdcfg->erase_value, bdcfg->erase_cycles, + bdcfg->badblock_behavior, bdcfg->power_cycles, + bdcfg->buffer, bdcfg->wear_buffer); + lfs_testbd_t *bd = cfg->context; + bd->cfg = bdcfg; + + // setup testing things + bd->persist = path; + bd->power_cycles = bd->cfg->power_cycles; + + if (bd->cfg->erase_cycles) { + if (bd->cfg->wear_buffer) { + bd->wear = bd->cfg->wear_buffer; + } else { + bd->wear = lfs_malloc(sizeof(lfs_testbd_wear_t) * cfg->block_count); + if (!bd->wear) { + LFS_TRACE("lfs_testbd_createcfg -> %d", LFS_ERR_NOMEM); + return LFS_ERR_NOMEM; + } + } + + memset(bd->wear, 0, sizeof(lfs_testbd_wear_t) * cfg->block_count); + } + + // create underlying block device + if (bd->persist) { + bd->u.file.cfg = (struct lfs_filebd_config){ + .erase_value = bd->cfg->erase_value, + }; + int err = lfs_filebd_createcfg(cfg, path, &bd->u.file.cfg); + LFS_TRACE("lfs_testbd_createcfg -> %d", err); + return err; + } else { + bd->u.ram.cfg = (struct lfs_rambd_config){ + .erase_value = bd->cfg->erase_value, + .buffer = bd->cfg->buffer, + }; + int err = lfs_rambd_createcfg(cfg, &bd->u.ram.cfg); + LFS_TRACE("lfs_testbd_createcfg -> %d", err); + return err; + } +} + +int lfs_testbd_create(const struct lfs_config *cfg, const char *path) { + LFS_TRACE("lfs_testbd_create(%p {.context=%p, " + ".read=%p, .prog=%p, .erase=%p, .sync=%p, " + ".read_size=%"PRIu32", .prog_size=%"PRIu32", " + ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " + "\"%s\")", + (void*)cfg, cfg->context, + (void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog, + (void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync, + cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count, + path); + static const struct lfs_testbd_config defaults = {.erase_value=-1}; + int err = lfs_testbd_createcfg(cfg, path, &defaults); + LFS_TRACE("lfs_testbd_create -> %d", err); + return err; +} + +int lfs_testbd_destroy(const struct lfs_config *cfg) { + LFS_TRACE("lfs_testbd_destroy(%p)", (void*)cfg); + lfs_testbd_t *bd = cfg->context; + if (bd->cfg->erase_cycles && !bd->cfg->wear_buffer) { + lfs_free(bd->wear); + } + + if (bd->persist) { + int err = lfs_filebd_destroy(cfg); + LFS_TRACE("lfs_testbd_destroy -> %d", err); + return err; + } else { + int err = lfs_rambd_destroy(cfg); + LFS_TRACE("lfs_testbd_destroy -> %d", err); + return err; + } +} + +/// Internal mapping to block devices /// +static int lfs_testbd_rawread(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size) { + lfs_testbd_t *bd = cfg->context; + if (bd->persist) { + return lfs_filebd_read(cfg, block, off, buffer, size); + } else { + return lfs_rambd_read(cfg, block, off, buffer, size); + } +} + +static int lfs_testbd_rawprog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size) { + lfs_testbd_t *bd = cfg->context; + if (bd->persist) { + return lfs_filebd_prog(cfg, block, off, buffer, size); + } else { + return lfs_rambd_prog(cfg, block, off, buffer, size); + } +} + +static int lfs_testbd_rawerase(const struct lfs_config *cfg, + lfs_block_t block) { + lfs_testbd_t *bd = cfg->context; + if (bd->persist) { + return lfs_filebd_erase(cfg, block); + } else { + return lfs_rambd_erase(cfg, block); + } +} + +static int lfs_testbd_rawsync(const struct lfs_config *cfg) { + lfs_testbd_t *bd = cfg->context; + if (bd->persist) { + return lfs_filebd_sync(cfg); + } else { + return lfs_rambd_sync(cfg); + } +} + +/// block device API /// +int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size) { + LFS_TRACE("lfs_testbd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + (void*)cfg, block, off, buffer, size); + lfs_testbd_t *bd = cfg->context; + + // check if read is valid + LFS_ASSERT(off % cfg->read_size == 0); + LFS_ASSERT(size % cfg->read_size == 0); + LFS_ASSERT(block < cfg->block_count); + + // block bad? + if (bd->cfg->erase_cycles && + bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_NOREAD && + bd->wear[block] >= bd->cfg->erase_cycles) { + LFS_TRACE("lfs_testbd_read -> %d", LFS_ERR_CORRUPT); + return LFS_ERR_CORRUPT; + } + + // read + int err = lfs_testbd_rawread(cfg, block, off, buffer, size); + LFS_TRACE("lfs_testbd_read -> %d", err); + return err; +} + +int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size) { + LFS_TRACE("lfs_testbd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + (void*)cfg, block, off, buffer, size); + lfs_testbd_t *bd = cfg->context; + + // check if write is valid + LFS_ASSERT(off % cfg->prog_size == 0); + LFS_ASSERT(size % cfg->prog_size == 0); + LFS_ASSERT(block < cfg->block_count); + + // block bad? + if (bd->cfg->erase_cycles && + bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_NOPROG && + bd->wear[block] >= bd->cfg->erase_cycles) { + LFS_TRACE("lfs_testbd_prog -> %d", LFS_ERR_CORRUPT); + return LFS_ERR_CORRUPT; + } + + // prog + int err = lfs_testbd_rawprog(cfg, block, off, buffer, size); + if (err) { + LFS_TRACE("lfs_testbd_prog -> %d", err); + return err; + } + + // lose power? + if (bd->power_cycles > 0) { + bd->power_cycles -= 1; + if (bd->power_cycles == 0) { + // sync to make sure we persist the last changes + lfs_testbd_rawsync(cfg); + // simulate power loss + exit(33); + } + } + + LFS_TRACE("lfs_testbd_prog -> %d", 0); + return 0; +} + +int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { + LFS_TRACE("lfs_testbd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); + lfs_testbd_t *bd = cfg->context; + + // check if erase is valid + LFS_ASSERT(block < cfg->block_count); + + // block bad? + if (bd->cfg->erase_cycles) { + if (bd->wear[block] >= bd->cfg->erase_cycles) { + if (bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_NOERASE) { + LFS_TRACE("lfs_testbd_erase -> %d", LFS_ERR_CORRUPT); + return LFS_ERR_CORRUPT; + } + } else { + // mark wear + bd->wear[block] += 1; + } + } + + // erase + int err = lfs_testbd_rawerase(cfg, block); + if (err) { + LFS_TRACE("lfs_testbd_erase -> %d", err); + return err; + } + + // lose power? + if (bd->power_cycles > 0) { + bd->power_cycles -= 1; + if (bd->power_cycles == 0) { + // sync to make sure we persist the last changes + lfs_testbd_rawsync(cfg); + // simulate power loss + exit(33); + } + } + + LFS_TRACE("lfs_testbd_prog -> %d", 0); + return 0; +} + +int lfs_testbd_sync(const struct lfs_config *cfg) { + LFS_TRACE("lfs_testbd_sync(%p)", (void*)cfg); + int err = lfs_testbd_rawsync(cfg); + LFS_TRACE("lfs_testbd_sync -> %d", err); + return err; +} + + +/// simulated wear operations /// +lfs_testbd_swear_t lfs_testbd_getwear(const struct lfs_config *cfg, + lfs_block_t block) { + LFS_TRACE("lfs_testbd_getwear(%p, %"PRIu32")", (void*)cfg, block); + lfs_testbd_t *bd = cfg->context; + + // check if block is valid + LFS_ASSERT(bd->cfg->erase_cycles); + LFS_ASSERT(block < cfg->block_count); + + LFS_TRACE("lfs_testbd_getwear -> %"PRIu32, bd->wear[block]); + return bd->wear[block]; +} + +int lfs_testbd_setwear(const struct lfs_config *cfg, + lfs_block_t block, lfs_testbd_wear_t wear) { + LFS_TRACE("lfs_testbd_setwear(%p, %"PRIu32")", (void*)cfg, block); + lfs_testbd_t *bd = cfg->context; + + // check if block is valid + LFS_ASSERT(bd->cfg->erase_cycles); + LFS_ASSERT(block < cfg->block_count); + + bd->wear[block] = wear; + + LFS_TRACE("lfs_testbd_setwear -> %d", 0); + return 0; +} diff --git a/testbd/lfs_testbd.h b/testbd/lfs_testbd.h new file mode 100644 index 00000000..9235457d --- /dev/null +++ b/testbd/lfs_testbd.h @@ -0,0 +1,130 @@ +/* + * Testing block device, wraps filebd and rambd while providing a bunch + * of hooks for testing littlefs in various conditions. + * + * Copyright (c) 2017, Arm Limited. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + */ +#ifndef LFS_TESTBD_H +#define LFS_TESTBD_H + +#include "lfs.h" +#include "lfs_util.h" +#include "rambd/lfs_rambd.h" +#include "filebd/lfs_filebd.h" + +#ifdef __cplusplus +extern "C" +{ +#endif + + +// Mode determining how "bad blocks" behave during testing. This +// simulates some real-world circumstances such as writes not +// going through (noprog), erases not sticking (noerase), and ECC +// failures (noread). +enum lfs_testbd_badblock_behavior { + LFS_TESTBD_BADBLOCK_NOPROG = 0, + LFS_TESTBD_BADBLOCK_NOERASE = 1, + LFS_TESTBD_BADBLOCK_NOREAD = 2, +}; + +// Type for measuring wear +typedef uint32_t lfs_testbd_wear_t; +typedef int32_t lfs_testbd_swear_t; + +// testbd config, this is required for testing +struct lfs_testbd_config { + // 8-bit erase value to use for simulating erases. -1 does not simulate + // erases, which can speed up testing by avoiding all the extra block-device + // operations to store the erase value. + int32_t erase_value; + + // Number of erase cycles before a block becomes "bad". The exact behavior + // of bad blocks is controlled by the badblock_mode. + uint32_t erase_cycles; + + // The mode determining how bad blocks fail + uint8_t badblock_behavior; + + // Number of write operations (erase/prog) before forcefully killing + // the program with exit. Simulates power-loss. 0 disables. + uint32_t power_cycles; + + // Optional buffer for RAM block device. + void *buffer; + + // Optional buffer for wear + void *wear_buffer; +}; + +// testbd state +typedef struct lfs_testbd { + union { + struct { + lfs_filebd_t bd; + struct lfs_filebd_config cfg; + } file; + struct { + lfs_rambd_t bd; + struct lfs_rambd_config cfg; + } ram; + } u; + + bool persist; + uint32_t power_cycles; + lfs_testbd_wear_t *wear; + + const struct lfs_testbd_config *cfg; +} lfs_testbd_t; + + +/// Block device API /// + +// Create a test block device using the geometry in lfs_config +// +// Note that filebd is used if a path is provided, if path is NULL +// testbd will use rambd which can be much faster. +int lfs_testbd_create(const struct lfs_config *cfg, const char *path); +int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, + const struct lfs_testbd_config *bdcfg); + +// Clean up memory associated with block device +int lfs_testbd_destroy(const struct lfs_config *cfg); + +// Read a block +int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, void *buffer, lfs_size_t size); + +// Program a block +// +// The block must have previously been erased. +int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, + lfs_off_t off, const void *buffer, lfs_size_t size); + +// Erase a block +// +// A block must be erased before being programmed. The +// state of an erased block is undefined. +int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block); + +// Sync the block device +int lfs_testbd_sync(const struct lfs_config *cfg); + + +/// Additional extended API for driving test features /// + +// Get simulated wear on a given block +lfs_testbd_swear_t lfs_testbd_getwear(const struct lfs_config *cfg, + lfs_block_t block); + +// Manually set simulated wear on a given block +int lfs_testbd_setwear(const struct lfs_config *cfg, + lfs_block_t block, lfs_testbd_wear_t wear); + + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif diff --git a/tests_/test_alloc.toml b/tests_/test_alloc.toml index ce80411a..8bfb5edc 100644 --- a/tests_/test_alloc.toml +++ b/tests_/test_alloc.toml @@ -503,6 +503,8 @@ code = ''' lfs_file_write(&lfs, &file, buffer, size) => size; } lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; ''' [[case]] # outdated lookahead and split dir test diff --git a/tests_/test_attrs.toml b/tests_/test_attrs.toml index f1254183..db8d0c7e 100644 --- a/tests_/test_attrs.toml +++ b/tests_/test_attrs.toml @@ -85,8 +85,8 @@ code = ''' lfs_file_write(&lfs, &file, "hello", strlen("hello")) => strlen("hello"); lfs_file_close(&lfs, &file); lfs_unmount(&lfs) => 0; - lfs_mount(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; memset(buffer, 0, sizeof(buffer)); lfs_setattr(&lfs, "/", 'A', "aaaa", 4) => 0; lfs_setattr(&lfs, "/", 'B', "bbbbbb", 6) => 0; @@ -162,7 +162,6 @@ code = ''' lfs_file_write(&lfs, &file, "hello", strlen("hello")) => strlen("hello"); lfs_file_close(&lfs, &file); lfs_unmount(&lfs) => 0; - lfs_mount(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; memset(buffer, 0, sizeof(buffer)); diff --git a/tests_/test_badblocks.toml b/tests_/test_badblocks.toml index 7599647c..7969d435 100644 --- a/tests_/test_badblocks.toml +++ b/tests_/test_badblocks.toml @@ -1,50 +1,12 @@ -# special bad-block block device hook for simulating blocks -# that are no longer writable -code = ''' -lfs_block_t *bbbd_badblocks; -size_t bbbd_badblocks_count; - -int bbbd_read(const struct lfs_config *c, lfs_block_t block, - lfs_off_t off, void *buffer, lfs_size_t size) { - for (size_t i = 0; i < bbbd_badblocks_count; i++) { - if (bbbd_badblocks[i] == block) { - return LFS_ERR_CORRUPT; - } - } - - return lfs_testbd_read(c, block, off, buffer, size); -} - -int bbbd_prog(const struct lfs_config *c, lfs_block_t block, - lfs_off_t off, const void *buffer, lfs_size_t size) { - for (size_t i = 0; i < bbbd_badblocks_count; i++) { - if (bbbd_badblocks[i] == block) { - return LFS_ERR_CORRUPT; - } - } - - return lfs_testbd_prog(c, block, off, buffer, size); -} - -int bbbd_erase(const struct lfs_config *c, lfs_block_t block) { - for (size_t i = 0; i < bbbd_badblocks_count; i++) { - if (bbbd_badblocks[i] == block) { - return LFS_ERR_CORRUPT; - } - } - - return lfs_testbd_erase(c, block); -} -''' - [[case]] # single bad blocks -define.LFS_BD_PROG = 'bbbd_prog' +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_NOPROG' define.NAMEMULT = 64 define.FILEMULT = 1 code = ''' for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { - bbbd_badblocks = &(lfs_block_t){badblock}; - bbbd_badblocks_count = 1; + lfs_testbd_setwear(&cfg, badblock-1, 0) => 0; + lfs_testbd_setwear(&cfg, badblock, 0xffffffff) => 0; lfs_format(&lfs, &cfg) => 0; @@ -103,13 +65,14 @@ code = ''' ''' [[case]] # single persistent blocks (can't erase) -define.LFS_BD_ERASE = 'bbbd_erase' +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_NOERASE' define.NAMEMULT = 64 define.FILEMULT = 1 code = ''' for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { - bbbd_badblocks = &(lfs_block_t){badblock}; - bbbd_badblocks_count = 1; + lfs_testbd_setwear(&cfg, badblock-1, 0) => 0; + lfs_testbd_setwear(&cfg, badblock, 0xffffffff) => 0; lfs_format(&lfs, &cfg) => 0; @@ -168,13 +131,14 @@ code = ''' ''' [[case]] # single unreadable blocks (can't read) -define.LFS_BD_READ = 'bbbd_read' +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_NOREAD' define.NAMEMULT = 64 define.FILEMULT = 1 code = ''' - for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { - bbbd_badblocks = &(lfs_block_t){badblock}; - bbbd_badblocks_count = 1; + for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT/2; badblock++) { + lfs_testbd_setwear(&cfg, badblock-1, 0) => 0; + lfs_testbd_setwear(&cfg, badblock, 0xffffffff) => 0; lfs_format(&lfs, &cfg) => 0; @@ -233,17 +197,17 @@ code = ''' ''' [[case]] # region corruption (causes cascading failures) -define.LFS_BD_PROG = '"BADTYPE == 0 ? bbbd_prog : lfs_testbd_prog "' -define.LFS_BD_ERASE = '"BADTYPE == 1 ? bbbd_erase : lfs_testbd_erase"' -define.LFS_BD_READ = '"BADTYPE == 2 ? bbbd_read : lfs_testbd_read "' -define.BADTYPE = 'range(3)' +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] define.NAMEMULT = 64 define.FILEMULT = 1 code = ''' - bbbd_badblocks_count = LFS_BLOCK_COUNT/2; - bbbd_badblocks = malloc(bbbd_badblocks_count*sizeof(lfs_block_t)); - for (size_t i = 0; i < bbbd_badblocks_count; i++) { - bbbd_badblocks[i] = i+2; + for (lfs_block_t i = 0; i < (LFS_BLOCK_COUNT-2)/2; i++) { + lfs_testbd_setwear(&cfg, i+2, 0xffffffff) => 0; } lfs_format(&lfs, &cfg) => 0; @@ -299,22 +263,20 @@ code = ''' lfs_file_close(&lfs, &file) => 0; } lfs_unmount(&lfs) => 0; - - free(bbbd_badblocks); ''' [[case]] # alternating corruption (causes cascading failures) -define.LFS_BD_PROG = '"BADTYPE == 0 ? bbbd_prog : lfs_testbd_prog "' -define.LFS_BD_ERASE = '"BADTYPE == 1 ? bbbd_erase : lfs_testbd_erase"' -define.LFS_BD_READ = '"BADTYPE == 2 ? bbbd_read : lfs_testbd_read "' -define.BADTYPE = 'range(3)' +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] define.NAMEMULT = 64 define.FILEMULT = 1 code = ''' - bbbd_badblocks_count = LFS_BLOCK_COUNT/2; - bbbd_badblocks = malloc(bbbd_badblocks_count*sizeof(lfs_block_t)); - for (size_t i = 0; i < bbbd_badblocks_count; i++) { - bbbd_badblocks[i] = (2*i) + 2; + for (lfs_block_t i = 0; i < (LFS_BLOCK_COUNT-2)/2; i++) { + lfs_testbd_setwear(&cfg, (2*i) + 2, 0xffffffff) => 0; } lfs_format(&lfs, &cfg) => 0; @@ -370,6 +332,20 @@ code = ''' lfs_file_close(&lfs, &file) => 0; } lfs_unmount(&lfs) => 0; +''' + +# other corner cases +[[case]] # bad superblocks (corrupt 1 or 0) +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] +code = ''' + lfs_testbd_setwear(&cfg, 0, 0xffffffff) => 0; + lfs_testbd_setwear(&cfg, 1, 0xffffffff) => 0; - free(bbbd_badblocks); + lfs_format(&lfs, &cfg) => LFS_ERR_NOSPC; + lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; ''' diff --git a/tests_/test_exhaustion.toml b/tests_/test_exhaustion.toml new file mode 100644 index 00000000..8dd3d150 --- /dev/null +++ b/tests_/test_exhaustion.toml @@ -0,0 +1,341 @@ +[[case]] # test running a filesystem to exhaustion +define.LFS_ERASE_CYCLES = 10 +define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] +define.FILES = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "roadrunner") => 0; + lfs_unmount(&lfs) => 0; + + uint32_t cycle = 0; + while (true) { + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // chose name, roughly random seed, and random 2^n size + sprintf(path, "roadrunner/test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); + assert(res == 1 || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + if (err == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "roadrunner/test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + char r; + lfs_file_read(&lfs, &file, &r, 1) => 1; + assert(r == c); + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + cycle += 1; + } + +exhausted: + // should still be readable + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "roadrunner/test%d", i); + lfs_stat(&lfs, path, &info) => 0; + } + lfs_unmount(&lfs) => 0; + + LFS_WARN("completed %d cycles", cycle); +''' + +[[case]] # test running a filesystem to exhaustion + # which also requires expanding superblocks +define.LFS_ERASE_CYCLES = 10 +define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] +define.FILES = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + + uint32_t cycle = 0; + while (true) { + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // chose name, roughly random seed, and random 2^n size + sprintf(path, "test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); + assert(res == 1 || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + if (err == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + char r; + lfs_file_read(&lfs, &file, &r, 1) => 1; + assert(r == c); + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + cycle += 1; + } + +exhausted: + // should still be readable + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "test%d", i); + lfs_stat(&lfs, path, &info) => 0; + } + lfs_unmount(&lfs) => 0; + + LFS_WARN("completed %d cycles", cycle); +''' + +# These are a sort of high-level litmus test for wear-leveling. One definition +# of wear-leveling is that increasing a block device's space translates directly +# into increasing the block devices lifetime. This is something we can actually +# check for. + +[[case]] # wear-level test running a filesystem to exhaustion +define.LFS_ERASE_CYCLES = 10 +define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] +define.FILES = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "roadrunner") => 0; + lfs_unmount(&lfs) => 0; + + uint32_t run_cycles[2]; + const uint32_t run_block_count[2] = {LFS_BLOCK_COUNT/2, LFS_BLOCK_COUNT}; + + for (int run = 0; run < 2; run++) { + for (lfs_block_t b = 0; b < LFS_BLOCK_COUNT; b++) { + lfs_testbd_setwear(&cfg, b, + (b < run_block_count[run]) ? 0 : LFS_ERASE_CYCLES) => 0; + } + + uint32_t cycle = 0; + while (true) { + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // chose name, roughly random seed, and random 2^n size + sprintf(path, "roadrunner/test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); + assert(res == 1 || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + if (err == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "roadrunner/test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + char r; + lfs_file_read(&lfs, &file, &r, 1) => 1; + assert(r == c); + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + cycle += 1; + } + +exhausted: + // should still be readable + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "roadrunner/test%d", i); + lfs_stat(&lfs, path, &info) => 0; + } + lfs_unmount(&lfs) => 0; + + run_cycles[run] = cycle; + LFS_WARN("completed %d blocks %d cycles", + run_block_count[run], run_cycles[run]); + } + + // check we increased the lifetime by 2x with ~5% error + LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/20); +''' + +[[case]] # wear-level test + expanding superblock +define.LFS_ERASE_CYCLES = 10 +define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_TESTBD_BADBLOCK_NOERASE', + 'LFS_TESTBD_BADBLOCK_NOREAD', +] +define.FILES = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + + uint32_t run_cycles[2]; + const uint32_t run_block_count[2] = {LFS_BLOCK_COUNT/2, LFS_BLOCK_COUNT}; + + for (int run = 0; run < 2; run++) { + for (lfs_block_t b = 0; b < LFS_BLOCK_COUNT; b++) { + lfs_testbd_setwear(&cfg, b, + (b < run_block_count[run]) ? 0 : LFS_ERASE_CYCLES) => 0; + } + + uint32_t cycle = 0; + while (true) { + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // chose name, roughly random seed, and random 2^n size + sprintf(path, "test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); + assert(res == 1 || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + if (err == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "test%d", i); + srand(cycle * i); + size = 1 << ((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + char r; + lfs_file_read(&lfs, &file, &r, 1) => 1; + assert(r == c); + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + cycle += 1; + } + +exhausted: + // should still be readable + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "test%d", i); + lfs_stat(&lfs, path, &info) => 0; + } + lfs_unmount(&lfs) => 0; + + run_cycles[run] = cycle; + LFS_WARN("completed %d blocks %d cycles", + run_block_count[run], run_cycles[run]); + } + + // check we increased the lifetime by 2x with ~5% error + LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/20); +''' diff --git a/tests_/test_format.toml b/tests_/test_format.toml index 702c0393..7932d54b 100644 --- a/tests_/test_format.toml +++ b/tests_/test_format.toml @@ -26,8 +26,6 @@ code = ''' lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; ''' -# TODO invalid superblock? (corrupt 1, 0) - [[case]] # expanding superblock define.BLOCK_CYCLES = [32, 33, 1] define.N = [10, 100, 1000] diff --git a/tests_/test_relocations.toml b/tests_/test_relocations.toml new file mode 100644 index 00000000..1fa73e51 --- /dev/null +++ b/tests_/test_relocations.toml @@ -0,0 +1,143 @@ +# specific corner cases worth explicitly testing for + +[[case]] # dangling split dir test +define.ITERATIONS = 20 +define.COUNT = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + // fill up filesystem so only ~16 blocks are left + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "padding", LFS_O_CREAT | LFS_O_WRONLY) => 0; + memset(buffer, 0, 512); + while (LFS_BLOCK_COUNT - lfs_fs_size(&lfs) > 16) { + lfs_file_write(&lfs, &file, buffer, 512) => 512; + } + lfs_file_close(&lfs, &file) => 0; + // make a child dir to use in bounded space + lfs_mkdir(&lfs, "child") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int j = 0; j < ITERATIONS; j++) { + for (int i = 0; i < COUNT; i++) { + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_close(&lfs, &file) => 0; + } + + lfs_dir_open(&lfs, &dir, "child") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + lfs_dir_read(&lfs, &dir, &info) => 1; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "test%03d_loooooooooooooooooong_name", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + strcmp(info.name, path) => 0; + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + if (j == ITERATIONS-1) { + break; + } + + for (int i = 0; i < COUNT; i++) { + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_remove(&lfs, path) => 0; + } + } + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + lfs_dir_open(&lfs, &dir, "child") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + lfs_dir_read(&lfs, &dir, &info) => 1; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "test%03d_loooooooooooooooooong_name", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + strcmp(info.name, path) => 0; + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_remove(&lfs, path) => 0; + } + lfs_unmount(&lfs) => 0; +''' + +[[case]] # outdated head test +define.ITERATIONS = 20 +define.COUNT = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + // fill up filesystem so only ~16 blocks are left + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "padding", LFS_O_CREAT | LFS_O_WRONLY) => 0; + memset(buffer, 0, 512); + while (LFS_BLOCK_COUNT - lfs_fs_size(&lfs) > 16) { + lfs_file_write(&lfs, &file, buffer, 512) => 512; + } + lfs_file_close(&lfs, &file) => 0; + // make a child dir to use in bounded space + lfs_mkdir(&lfs, "child") => 0; + lfs_unmount(&lfs) => 0; + + lfs_mount(&lfs, &cfg) => 0; + for (int j = 0; j < ITERATIONS; j++) { + for (int i = 0; i < COUNT; i++) { + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; + lfs_file_close(&lfs, &file) => 0; + } + + lfs_dir_open(&lfs, &dir, "child") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + lfs_dir_read(&lfs, &dir, &info) => 1; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "test%03d_loooooooooooooooooong_name", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + strcmp(info.name, path) => 0; + info.size => 0; + + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_file_open(&lfs, &file, path, LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hi", 2) => 2; + lfs_file_close(&lfs, &file) => 0; + } + lfs_dir_read(&lfs, &dir, &info) => 0; + + lfs_dir_rewind(&lfs, &dir) => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + lfs_dir_read(&lfs, &dir, &info) => 1; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "test%03d_loooooooooooooooooong_name", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + strcmp(info.name, path) => 0; + info.size => 2; + + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_file_open(&lfs, &file, path, LFS_O_WRONLY) => 0; + lfs_file_write(&lfs, &file, "hi", 2) => 2; + lfs_file_close(&lfs, &file) => 0; + } + lfs_dir_read(&lfs, &dir, &info) => 0; + + lfs_dir_rewind(&lfs, &dir) => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + lfs_dir_read(&lfs, &dir, &info) => 1; + for (int i = 0; i < COUNT; i++) { + sprintf(path, "test%03d_loooooooooooooooooong_name", i); + lfs_dir_read(&lfs, &dir, &info) => 1; + strcmp(info.name, path) => 0; + info.size => 2; + } + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + for (int i = 0; i < COUNT; i++) { + sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); + lfs_remove(&lfs, path) => 0; + } + } + lfs_unmount(&lfs) => 0; +''' From 9453ebd15d590778a5207920b2671a4fff91a9e0 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sat, 18 Jan 2020 20:20:43 -0600 Subject: [PATCH 10/41] Added/improved disk-reading debug scripts Also fixed a bug in dir splitting when there's a large number of open files, which was the main reason I was trying to make it easier to debug disk images. One part of the recent test changes was to move away from the file-per-block emubd and instead simulate storage with a single contiguous file. The file-per-block format was marginally useful at the beginning, but as the remaining bugs get more subtle, it becomes more useful to inspect littlefs through scripts that make the underlying metadata more human-readable. The key benefit of switching to a contiguous file is these same scripts can be reused for real disk images and can even read through /dev/sdb or similar. - ./scripts/readblock.py disk block_size block off data 00000000: 71 01 00 00 f0 0f ff f7 6c 69 74 74 6c 65 66 73 q.......littlefs 00000010: 2f e0 00 10 00 00 02 00 00 02 00 00 00 04 00 00 /............... 00000020: ff 00 00 00 ff ff ff 7f fe 03 00 00 20 00 04 19 ............... 00000030: 61 00 00 0c 00 62 20 30 0c 09 a0 01 00 00 64 00 a....b 0......d. ... readblock.py prints a hex dump of a given block on disk. It's basically just "dd if=disk bs=block_size count=1 skip=block | xxd -g1 -" but with less typing. - ./scripts/readmdir.py disk block_size block1 block2 off tag type id len data (truncated) 0000003b: 0020000a dir 0 10 63 6f 6c 64 63 6f 66 66 coldcoff 00000049: 20000008 dirstruct 0 8 02 02 00 00 03 02 00 00 ........ 00000008: 00200409 dir 1 9 68 6f 74 63 6f 66 66 65 hotcoffe 00000015: 20000408 dirstruct 1 8 fe 01 00 00 ff 01 00 00 ........ readmdir.py prints info about the tags in a metadata pair on disk. It can print the currently active tags as well as the raw log of the metadata pair. - ./scripts/readtree.py disk block_size superblock "littlefs" version v2.0 block_size 512 block_count 1024 name_max 255 file_max 2147483647 attr_max 1022 gstate 0x000000000000000000000000 dir "/" mdir {0x0, 0x1} rev 3 v id 0 superblock "littlefs" inline size 24 mdir {0x77, 0x78} rev 1 id 0 dir "coffee" dir {0x1fc, 0x1fd} dir "/coffee" mdir {0x1fd, 0x1fc} rev 2 id 0 dir "coldcoffee" dir {0x202, 0x203} id 1 dir "hotcoffee" dir {0x1fe, 0x1ff} dir "/coffee/coldcoffee" mdir {0x202, 0x203} rev 1 dir "/coffee/warmcoffee" mdir {0x200, 0x201} rev 1 readtree.py parses the littlefs tree and prints info about the semantics of what's on disk. This includes the superblock, global-state, and directories/metadata-pairs. It doesn't print the filesystem tree though, that could be a different tool. --- lfs.c | 2 +- scripts/readblock.py | 26 +++ scripts/readmdir.py | 332 ++++++++++++++++++++++++++++++++++ scripts/readtree.py | 273 ++++++++++++++++++++++++++++ scripts/test_.py | 4 +- tests_/test_exhaustion.toml | 8 +- tests_/test_interspersed.toml | 26 +-- tests_/test_seek.toml | 8 +- 8 files changed, 645 insertions(+), 34 deletions(-) create mode 100755 scripts/readblock.py create mode 100755 scripts/readmdir.py create mode 100755 scripts/readtree.py diff --git a/lfs.c b/lfs.c index 2758c29c..e26d7451 100644 --- a/lfs.c +++ b/lfs.c @@ -1825,7 +1825,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, // for things that are for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) { if (lfs_pair_cmp(d->m.pair, copy.pair) == 0) { - d->m = *dir; + d->m = copy; if (d->id == lfs_tag_id(deletetag)) { d->m.pair[0] = LFS_BLOCK_NULL; d->m.pair[1] = LFS_BLOCK_NULL; diff --git a/scripts/readblock.py b/scripts/readblock.py new file mode 100755 index 00000000..817517bc --- /dev/null +++ b/scripts/readblock.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 + +import subprocess as sp + +def main(args): + with open(args.disk, 'rb') as f: + f.seek(args.block * args.block_size) + block = (f.read(args.block_size) + .ljust(args.block_size, b'\xff')) + + # what did you expect? + print("%-8s %-s" % ('off', 'data')) + return sp.run(['xxd', '-g1', '-'], input=block).returncode + +if __name__ == "__main__": + import argparse + import sys + parser = argparse.ArgumentParser( + description="Hex dump a specific block in a disk.") + parser.add_argument('disk', + help="File representing the block device.") + parser.add_argument('block_size', type=lambda x: int(x, 0), + help="Size of a block in bytes.") + parser.add_argument('block', type=lambda x: int(x, 0), + help="Address of block to dump.") + sys.exit(main(parser.parse_args())) diff --git a/scripts/readmdir.py b/scripts/readmdir.py new file mode 100755 index 00000000..a60acbe7 --- /dev/null +++ b/scripts/readmdir.py @@ -0,0 +1,332 @@ +#!/usr/bin/env python3 + +import struct +import binascii +import itertools as it + +TAG_TYPES = { + 'splice': (0x700, 0x400), + 'create': (0x7ff, 0x401), + 'delete': (0x7ff, 0x4ff), + 'name': (0x700, 0x000), + 'reg': (0x7ff, 0x001), + 'dir': (0x7ff, 0x002), + 'superblock': (0x7ff, 0x0ff), + 'struct': (0x700, 0x200), + 'dirstruct': (0x7ff, 0x200), + 'ctzstruct': (0x7ff, 0x202), + 'inlinestruct': (0x7ff, 0x201), + 'userattr': (0x700, 0x300), + 'tail': (0x700, 0x600), + 'softtail': (0x7ff, 0x600), + 'hardtail': (0x7ff, 0x601), + 'gstate': (0x700, 0x700), + 'movestate': (0x7ff, 0x7ff), + 'crc': (0x700, 0x500), +} + +class Tag: + def __init__(self, *args): + if len(args) == 1: + self.tag = args[0] + elif len(args) == 3: + if isinstance(args[0], str): + type = TAG_TYPES[args[0]][1] + else: + type = args[0] + + if isinstance(args[1], str): + id = int(args[1], 0) if args[1] not in 'x.' else 0x3ff + else: + id = args[1] + + if isinstance(args[2], str): + size = int(args[2], str) if args[2] not in 'x.' else 0x3ff + else: + size = args[2] + + self.tag = (type << 20) | (id << 10) | size + else: + assert False + + @property + def isvalid(self): + return not bool(self.tag & 0x80000000) + + @property + def isattr(self): + return not bool(self.tag & 0x40000000) + + @property + def iscompactable(self): + return bool(self.tag & 0x20000000) + + @property + def isunique(self): + return not bool(self.tag & 0x10000000) + + @property + def type(self): + return (self.tag & 0x7ff00000) >> 20 + + @property + def type1(self): + return (self.tag & 0x70000000) >> 20 + + @property + def type3(self): + return (self.tag & 0x7ff00000) >> 20 + + @property + def id(self): + return (self.tag & 0x000ffc00) >> 10 + + @property + def size(self): + return (self.tag & 0x000003ff) >> 0 + + @property + def dsize(self): + return 4 + (self.size if self.size != 0x3ff else 0) + + @property + def chunk(self): + return self.type & 0xff + + @property + def schunk(self): + return struct.unpack('b', struct.pack('B', self.chunk))[0] + + def is_(self, type): + return (self.type & TAG_TYPES[type][0]) == TAG_TYPES[type][1] + + def mkmask(self): + return Tag( + 0x700 if self.isunique else 0x7ff, + 0x3ff if self.isattr else 0, + 0) + + def typerepr(self): + if self.is_('crc') and getattr(self, 'crc', 0xffffffff) != 0xffffffff: + return 'crc (bad)' + + reverse_types = {v: k for k, v in TAG_TYPES.items()} + for prefix in range(12): + mask = 0x7ff & ~((1 << prefix)-1) + if (mask, self.type & mask) in reverse_types: + type = reverse_types[mask, self.type & mask] + if prefix > 0: + return '%s %#0*x' % ( + type, prefix//4, self.type & ((1 << prefix)-1)) + else: + return type + else: + return '%02x' % self.type + + def idrepr(self): + return repr(self.id) if self.id != 0x3ff else '.' + + def sizerepr(self): + return repr(self.size) if self.size != 0x3ff else 'x' + + def __repr__(self): + return 'Tag(%r, %d, %d)' % (self.typerepr(), self.id, self.size) + + def __lt__(self, other): + return (self.id, self.type) < (other.id, other.type) + + def __bool__(self): + return self.isvalid + + def __int__(self): + return self.tag + + def __index__(self): + return self.tag + +class MetadataPair: + def __init__(self, blocks): + if len(blocks) > 1: + self.pair = [MetadataPair([block]) for block in blocks] + self.pair = sorted(self.pair, reverse=True) + + self.data = self.pair[0].data + self.rev = self.pair[0].rev + self.tags = self.pair[0].tags + self.ids = self.pair[0].ids + self.log = self.pair[0].log + self.all_ = self.pair[0].all_ + return + + self.pair = [self] + self.data = blocks[0] + block = self.data + + self.rev, = struct.unpack('= 4: + ntag, = struct.unpack('>I', block[off:off+4]) + + tag = Tag(int(tag) ^ ntag) + tag.off = off + 4 + tag.data = block[off+4:off+tag.dsize] + if tag.is_('crc'): + crc = binascii.crc32(block[off:off+4+4], crc) + else: + crc = binascii.crc32(block[off:off+tag.dsize], crc) + tag.crc = crc + off += tag.dsize + + self.all_.append(tag) + + if tag.is_('crc'): + # is valid commit? + if crc != 0xffffffff: + corrupt = True + if not corrupt: + self.log = self.all_.copy() + + # reset tag parsing + crc = 0 + tag = Tag(int(tag) ^ ((tag.type & 1) << 31)) + + # find most recent tags + self.tags = [] + for tag in self.log: + if tag.is_('crc') or tag.is_('splice'): + continue + + if tag in self and self[tag] is tag: + self.tags.append(tag) + + self.tags = sorted(self.tags) + + # and ids + self.ids = list(it.takewhile( + lambda id: Tag('name', id, 0) in self, + it.count())) + + def __bool__(self): + return bool(self.log) + + def __lt__(self, other): + # corrupt blocks don't count + if not self and other: + return True + + # use sequence arithmetic to avoid overflow + return not ((other.rev - self.rev) & 0x80000000) + + def __contains__(self, args): + try: + self[args] + return True + except KeyError: + return False + + def __getitem__(self, args): + if isinstance(args, tuple): + gmask, gtag = args + else: + gmask, gtag = args.mkmask(), args + + gdiff = 0 + for tag in reversed(self.log): + if (gmask.id != 0 and tag.is_('splice') and + tag.id <= gtag.id - gdiff): + if tag.is_('create') and tag.id == gtag.id - gdiff: + # creation point + break + + gdiff += tag.schunk + + if (int(gmask) & int(tag)) == (int(gmask) & int( + Tag(gtag.type, gtag.id - gdiff, gtag.size))): + if tag.size == 0x3ff: + # deleted + break + + return tag + + raise KeyError(gmask, gtag) + + def _dump_tags(self, tags, truncate=True): + sys.stdout.write("%-8s %-8s %-13s %4s %4s %s\n" % ( + 'off', 'tag', 'type', 'id', 'len', + 'data (truncated)' if truncate else 12*' '+'data')) + + for tag in tags: + sys.stdout.write("%08x: %08x %-13s %4s %4s" % ( + tag.off, tag, + tag.typerepr(), tag.idrepr(), tag.sizerepr())) + if truncate: + sys.stdout.write(" %-23s %-8s\n" % ( + ' '.join('%02x' % c for c in tag.data[:8]), + ''.join(c if c >= ' ' and c <= '~' else '.' + for c in map(chr, tag.data[:8])))) + else: + sys.stdout.write("\n") + for i in range(0, len(tag.data), 16): + sys.stdout.write("%08x: %-47s %-16s\n" % ( + tag.off+i, + ' '.join('%02x' % c for c in tag.data[i:i+16]), + ''.join(c if c >= ' ' and c <= '~' else '.' + for c in map(chr, tag.data[i:i+16])))) + + def dump_tags(self, truncate=True): + self._dump_tags(self.tags, truncate=truncate) + + def dump_log(self, truncate=True): + self._dump_tags(self.log, truncate=truncate) + + def dump_all(self, truncate=True): + self._dump_tags(self.all_, truncate=truncate) + +def main(args): + blocks = [] + with open(args.disk, 'rb') as f: + for block in [args.block1, args.block2]: + if block is None: + continue + f.seek(block * args.block_size) + blocks.append(f.read(args.block_size) + .ljust(args.block_size, b'\xff')) + + # find most recent pair + mdir = MetadataPair(blocks) + if args.all: + mdir.dump_all(truncate=not args.no_truncate) + elif args.log: + mdir.dump_log(truncate=not args.no_truncate) + else: + mdir.dump_tags(truncate=not args.no_truncate) + + return 0 if mdir else 1 + +if __name__ == "__main__": + import argparse + import sys + parser = argparse.ArgumentParser( + description="Dump useful info about metadata pairs in littlefs.") + parser.add_argument('disk', + help="File representing the block device.") + parser.add_argument('block_size', type=lambda x: int(x, 0), + help="Size of a block in bytes.") + parser.add_argument('block1', type=lambda x: int(x, 0), + help="First block address for finding the metadata pair.") + parser.add_argument('block2', nargs='?', type=lambda x: int(x, 0), + help="Second block address for finding the metadata pair.") + parser.add_argument('-a', '--all', action='store_true', + help="Show all tags in log, included tags in corrupted commits.") + parser.add_argument('-l', '--log', action='store_true', + help="Show tags in log.") + parser.add_argument('-T', '--no-truncate', action='store_true', + help="Don't truncate large amounts of data in tags.") + sys.exit(main(parser.parse_args())) diff --git a/scripts/readtree.py b/scripts/readtree.py new file mode 100755 index 00000000..2bae10e0 --- /dev/null +++ b/scripts/readtree.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 + +import struct +import sys +import json +import io +import itertools as it +from readmdir import Tag, MetadataPair + +def popc(x): + return bin(x).count('1') + +def ctz(x): + return len(bin(x)) - len(bin(x).rstrip('0')) + +def dumptags(args, mdir, f): + if args.all: + tags = mdir.all_ + elif args.log: + tags = mdir.log + else: + tags = mdir.tags + + for k, tag in enumerate(tags): + f.write("tag %08x %s" % (tag, tag.typerepr())) + if tag.id != 0x3ff: + f.write(" id %d" % tag.id) + if tag.size != 0x3ff: + f.write(" size %d" % tag.size) + if tag.is_('name'): + f.write(" name %s" % + json.dumps(tag.data.decode('utf8'))) + if tag.is_('dirstruct'): + f.write(" dir {%#x, %#x}" % struct.unpack( + '= ' ' and c <= '~' else '.' + for c in map(chr, tag.data[i:i+16])))) + +def dumpentries(args, mdir, f): + for k, id_ in enumerate(mdir.ids): + name = mdir[Tag('name', id_, 0)] + struct_ = mdir[Tag('struct', id_, 0)] + + f.write("id %d %s %s" % ( + name.id, name.typerepr(), + json.dumps(name.data.decode('utf8')))) + if struct_.is_('dirstruct'): + f.write(" dir {%#x, %#x}" % struct.unpack( + '= ' ' and c <= '~' else '.' + for c in map(chr, struct_.data[i:i+16])))) + elif args.data and struct_.is_('ctzstruct'): + block, size = struct.unpack( + '= 0: + f2.seek(block * args.block_size) + dat = f2.read(args.block_size) + data.append(dat[4*(ctz(i)+1) if i != 0 else 0:]) + block, = struct.unpack('= ' ' and c <= '~' else '.' + for c in map(chr, data[i:i+16])))) + + for tag in mdir.tags: + if tag.id==id_ and tag.is_('userattr'): + f.write("id %d %s size %d\n" % ( + id_, tag.typerepr(), tag.size)) + + if args.data: + for i in range(0, len(tag.data), 16): + f.write(" %-47s %-16s\n" % ( + ' '.join('%02x' % c for c in tag.data[i:i+16]), + ''.join(c if c >= ' ' and c <= '~' else '.' + for c in map(chr, tag.data[i:i+16])))) + +def main(args): + with open(args.disk, 'rb') as f: + dirs = [] + superblock = None + gstate = b'' + mdirs = [] + tail = (args.block1, args.block2) + hard = False + while True: + # load mdir + data = [] + blocks = {} + for block in tail: + f.seek(block * args.block_size) + data.append(f.read(args.block_size) + .ljust(args.block_size, b'\xff')) + blocks[id(data[-1])] = block + mdir = MetadataPair(data) + mdir.blocks = tuple(blocks[id(p.data)] for p in mdir.pair) + + # fetch some key metadata as a we scan + try: + mdir.tail = mdir[Tag('tail', 0, 0)] + if mdir.tail.size != 8 or mdir.tail.data == 8*b'\xff': + mdir.tail = None + except KeyError: + mdir.tail = None + + # have superblock? + try: + nsuperblock = mdir[ + Tag(0x7ff, 0x3ff, 0), Tag('superblock', 0, 0)] + superblock = nsuperblock, mdir[Tag('inlinestruct', 0, 0)] + except KeyError: + pass + + # have gstate? + try: + ngstate = mdir[Tag('movestate', 0, 0)] + gstate = bytes((a or 0) ^ (b or 0) + for a,b in it.zip_longest(gstate, ngstate.data)) + except KeyError: + pass + + # add to directories + mdirs.append(mdir) + if mdir.tail is None or not mdir.tail.is_('hardtail'): + dirs.append(mdirs) + mdirs = [] + + if mdir.tail is None: + break + + tail = struct.unpack(' 2*run_cycles[0]-run_cycles[0]/20); + // check we increased the lifetime by 2x with ~10% error + LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/10); ''' [[case]] # wear-level test + expanding superblock @@ -336,6 +336,6 @@ exhausted: run_block_count[run], run_cycles[run]); } - // check we increased the lifetime by 2x with ~5% error - LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/20); + // check we increased the lifetime by 2x with ~10% error + LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/10); ''' diff --git a/tests_/test_interspersed.toml b/tests_/test_interspersed.toml index 32f79e79..87a05780 100644 --- a/tests_/test_interspersed.toml +++ b/tests_/test_interspersed.toml @@ -1,16 +1,7 @@ [[case]] # interspersed file test -# TODO FILES=26 found bug -#define.SIZE = [10, 100] -#define.FILES = [4, 10, 26] -define = [ - {SIZE=10, FILES=4}, - {SIZE=10, FILES=10}, - #{SIZE=10, FILES=26}, - {SIZE=100, FILES=4}, - {SIZE=100, FILES=10}, - #{SIZE=100, FILES=26}, -] +define.SIZE = [10, 100] +define.FILES = [4, 10, 26] code = ''' lfs_file_t files[FILES]; const char alphas[] = "abcdefghijklmnopqrstuvwxyz"; @@ -182,17 +173,8 @@ code = ''' ''' [[case]] # reentrant interspersed file test -# TODO FILES=26 found bug -#define.SIZE = [10, 100] -#define.FILES = [4, 10, 26] -define = [ - {SIZE=10, FILES=4}, - {SIZE=10, FILES=10}, - #{SIZE=10, FILES=26}, - {SIZE=100, FILES=4}, - #{SIZE=100, FILES=10}, - #{SIZE=100, FILES=26}, -] +define.SIZE = [10, 100] +define.FILES = [4, 10, 26] reentrant = true code = ''' lfs_file_t files[FILES]; diff --git a/tests_/test_seek.toml b/tests_/test_seek.toml index 586ab719..79d7728a 100644 --- a/tests_/test_seek.toml +++ b/tests_/test_seek.toml @@ -24,14 +24,14 @@ code = ''' lfs_mount(&lfs, &cfg) => 0; lfs_file_open(&lfs, &file, "kitty", LFS_O_RDONLY) => 0; - lfs_soff_t pos; + lfs_soff_t pos = -1; size = strlen("kittycatcat"); for (int i = 0; i < SKIP; i++) { lfs_file_read(&lfs, &file, buffer, size) => size; memcmp(buffer, "kittycatcat", size) => 0; pos = lfs_file_tell(&lfs, &file); } - pos >= 0 => 1; + assert(pos >= 0); lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; lfs_file_read(&lfs, &file, buffer, size) => size; @@ -93,14 +93,14 @@ code = ''' lfs_mount(&lfs, &cfg) => 0; lfs_file_open(&lfs, &file, "kitty", LFS_O_RDWR) => 0; - lfs_soff_t pos; + lfs_soff_t pos = -1; size = strlen("kittycatcat"); for (int i = 0; i < SKIP; i++) { lfs_file_read(&lfs, &file, buffer, size) => size; memcmp(buffer, "kittycatcat", size) => 0; pos = lfs_file_tell(&lfs, &file); } - pos >= 0 => 1; + assert(pos >= 0); memcpy(buffer, "doggodogdog", size); lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; From f4b6a6b3284d8b721ba3db0bba93ee45fde7ead9 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Mon, 20 Jan 2020 17:35:45 -0600 Subject: [PATCH 11/41] Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain --- lfs.c | 225 ++++++----- scripts/readmdir.py | 34 +- scripts/readtree.py | 19 +- tests_/test_move.toml | 842 +++++++++++++++++++++++++++++++++++++++++- 4 files changed, 1003 insertions(+), 117 deletions(-) diff --git a/lfs.c b/lfs.c index e26d7451..6a0a9245 100644 --- a/lfs.c +++ b/lfs.c @@ -265,6 +265,12 @@ typedef int32_t lfs_stag_t; #define LFS_MKTAG(type, id, size) \ (((lfs_tag_t)(type) << 20) | ((lfs_tag_t)(id) << 10) | (lfs_tag_t)(size)) +#define LFS_MKTAG_IF(cond, type, id, size) \ + ((cond) ? LFS_MKTAG(type, id, size) : LFS_MKTAG(LFS_FROM_NOOP, 0, 0)) + +#define LFS_MKTAG_IF_ELSE(cond, type1, id1, size1, type2, id2, size2) \ + ((cond) ? LFS_MKTAG(type, id, size) : LFS_MKTAG(type2, id2, size2)) + static inline bool lfs_tag_isvalid(lfs_tag_t tag) { return !(tag & 0x80000000); } @@ -652,7 +658,7 @@ static int lfs_dir_traverse_filter(void *p, static int lfs_dir_traverse(lfs_t *lfs, const lfs_mdir_t *dir, lfs_off_t off, lfs_tag_t ptag, - const struct lfs_mattr *attrs, int attrcount, bool hasseenmove, + const struct lfs_mattr *attrs, int attrcount, lfs_tag_t tmask, lfs_tag_t ttag, uint16_t begin, uint16_t end, int16_t diff, int (*cb)(void *data, lfs_tag_t tag, const void *buffer), void *data) { @@ -680,13 +686,6 @@ static int lfs_dir_traverse(lfs_t *lfs, buffer = attrs[0].buffer; attrs += 1; attrcount -= 1; - } else if (!hasseenmove && - lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) { - // Wait, we have pending move? Handle this here (we need to - // or else we risk letting moves fall out of date) - tag = lfs->gpending.tag & LFS_MKTAG(0x7ff, 0x3ff, 0); - buffer = NULL; - hasseenmove = true; } else { return 0; } @@ -701,7 +700,7 @@ static int lfs_dir_traverse(lfs_t *lfs, if (lfs_tag_id(tmask) != 0) { // scan for duplicates and update tag based on creates/deletes int filter = lfs_dir_traverse(lfs, - dir, off, ptag, attrs, attrcount, hasseenmove, + dir, off, ptag, attrs, attrcount, 0, 0, 0, 0, 0, lfs_dir_traverse_filter, &tag); if (filter < 0) { @@ -725,7 +724,7 @@ static int lfs_dir_traverse(lfs_t *lfs, uint16_t fromid = lfs_tag_size(tag); uint16_t toid = lfs_tag_id(tag); int err = lfs_dir_traverse(lfs, - buffer, 0, LFS_BLOCK_NULL, NULL, 0, true, + buffer, 0, LFS_BLOCK_NULL, NULL, 0, LFS_MKTAG(0x600, 0x3ff, 0), LFS_MKTAG(LFS_TYPE_STRUCT, 0, 0), fromid, fromid+1, toid-fromid+diff, @@ -1443,7 +1442,7 @@ static int lfs_dir_compact(lfs_t *lfs, // find size lfs_size_t size = 0; int err = lfs_dir_traverse(lfs, - source, 0, LFS_BLOCK_NULL, attrs, attrcount, false, + source, 0, LFS_BLOCK_NULL, attrs, attrcount, LFS_MKTAG(0x400, 0x3ff, 0), LFS_MKTAG(LFS_TYPE_NAME, 0, 0), begin, end, -begin, @@ -1569,7 +1568,7 @@ static int lfs_dir_compact(lfs_t *lfs, // traverse the directory, this time writing out all unique tags err = lfs_dir_traverse(lfs, - source, 0, LFS_BLOCK_NULL, attrs, attrcount, false, + source, 0, LFS_BLOCK_NULL, attrs, attrcount, LFS_MKTAG(0x400, 0x3ff, 0), LFS_MKTAG(LFS_TYPE_NAME, 0, 0), begin, end, -begin, @@ -1627,11 +1626,6 @@ static int lfs_dir_compact(lfs_t *lfs, dir->count = end - begin; dir->off = commit.off; dir->etag = commit.ptag; - // note we able to have already handled move here - if (lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) { - lfs_gstate_xormove(&lfs->gpending, - &lfs->gpending, 0x3ff, NULL); - } } break; @@ -1696,16 +1690,14 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } // calculate changes to the directory - lfs_tag_t deletetag = LFS_BLOCK_NULL; - lfs_tag_t createtag = LFS_BLOCK_NULL; + bool hasdelete = false; for (int i = 0; i < attrcount; i++) { if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE) { - createtag = attrs[i].tag; dir->count += 1; } else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE) { - deletetag = attrs[i].tag; LFS_ASSERT(dir->count > 0); dir->count -= 1; + hasdelete = true; } else if (lfs_tag_type1(attrs[i].tag) == LFS_TYPE_TAIL) { dir->tail[0] = ((lfs_block_t*)attrs[i].buffer)[0]; dir->tail[1] = ((lfs_block_t*)attrs[i].buffer)[1]; @@ -1714,18 +1706,8 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } } - // do we have a pending move? - if (lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) { - deletetag = lfs->gpending.tag & LFS_MKTAG(0x7ff, 0x3ff, 0); - LFS_ASSERT(dir->count > 0); - dir->count -= 1; - - // mark gdelta so we reflect the move we will fix - lfs_gstate_xormove(&lfs->gdelta, &lfs->gpending, 0x3ff, NULL); - } - // should we actually drop the directory block? - if (lfs_tag_isvalid(deletetag) && dir->count == 0) { + if (hasdelete && dir->count == 0) { lfs_mdir_t pdir; int err = lfs_fs_pred(lfs, dir->pair, &pdir); if (err && err != LFS_ERR_NOENT) { @@ -1752,7 +1734,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, // traverse attrs that need to be written out lfs_pair_tole32(dir->tail); int err = lfs_dir_traverse(lfs, - dir, dir->off, dir->etag, attrs, attrcount, false, + dir, dir->off, dir->etag, attrs, attrcount, 0, 0, 0, 0, 0, lfs_dir_commit_commit, &(struct lfs_dir_commit_commit){ lfs, &commit}); @@ -1797,13 +1779,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0); dir->off = commit.off; dir->etag = commit.ptag; - - // note we able to have already handled move here - if (lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) { - lfs_gstate_xormove(&lfs->gpending, &lfs->gpending, 0x3ff, NULL); - } - - // update gstate + // and update gstate lfs->gstate = lfs->gpending; lfs->gdelta = (struct lfs_gstate){0}; } else { @@ -1818,29 +1794,42 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } } - // update any directories that are affected - lfs_mdir_t copy = *dir; - - // two passes, once for things that aren't us, and one - // for things that are + // this complicated bit of logic is for fixing up any active + // metadata-pairs that we may have affected + // + // note we have to make two passes since the mdir passed to + // lfs_dir_commit could also be in this list, and even then + // we need to copy the pair so they don't get clobbered if we refetch + // our mdir. for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) { - if (lfs_pair_cmp(d->m.pair, copy.pair) == 0) { - d->m = copy; - if (d->id == lfs_tag_id(deletetag)) { - d->m.pair[0] = LFS_BLOCK_NULL; - d->m.pair[1] = LFS_BLOCK_NULL; - } else if (d->id > lfs_tag_id(deletetag)) { - d->id -= 1; - if (d->type == LFS_TYPE_DIR) { - ((lfs_dir_t*)d)->pos -= 1; - } - } else if (&d->m != dir && d->id >= lfs_tag_id(createtag)) { - d->id += 1; - if (d->type == LFS_TYPE_DIR) { - ((lfs_dir_t*)d)->pos += 1; + if (&d->m != dir && lfs_pair_cmp(d->m.pair, dir->pair) == 0) { + d->m = *dir; + + for (int i = 0; i < attrcount; i++) { + if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE && + d->id == lfs_tag_id(attrs[i].tag)) { + d->m.pair[0] = LFS_BLOCK_NULL; + d->m.pair[1] = LFS_BLOCK_NULL; + } else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE && + d->id > lfs_tag_id(attrs[i].tag)) { + d->id -= 1; + if (d->type == LFS_TYPE_DIR) { + ((lfs_dir_t*)d)->pos -= 1; + } + } else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE && + d->id >= lfs_tag_id(attrs[i].tag)) { + d->id += 1; + if (d->type == LFS_TYPE_DIR) { + ((lfs_dir_t*)d)->pos += 1; + } } } + } + } + lfs_block_t pair[2] = {dir->pair[0], dir->pair[1]}; + for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) { + if (lfs_pair_cmp(d->m.pair, pair) == 0) { while (d->id >= d->m.count && d->m.split) { // we split and id is on tail now d->id -= d->m.count; @@ -1931,9 +1920,8 @@ int lfs_mkdir(lfs_t *lfs, const char *path) { {LFS_MKTAG(LFS_TYPE_CREATE, id, 0), NULL}, {LFS_MKTAG(LFS_TYPE_DIR, id, nlen), path}, {LFS_MKTAG(LFS_TYPE_DIRSTRUCT, id, 8), dir.pair}, - {!cwd.split - ? LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8) - : LFS_MKTAG(LFS_FROM_NOOP, 0, 0), dir.pair})); + {LFS_MKTAG_IF(!cwd.split, + LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir.pair})); lfs_pair_fromle32(dir.pair); if (err) { LFS_TRACE("lfs_mkdir -> %d", err); @@ -2375,9 +2363,9 @@ int lfs_file_opencfg(lfs_t *lfs, lfs_file_t *file, // get next slot and create entry to remember name err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS( - {LFS_MKTAG(LFS_TYPE_CREATE, file->id, 0), NULL}, + {LFS_MKTAG(LFS_TYPE_CREATE, file->id, 0)}, {LFS_MKTAG(LFS_TYPE_REG, file->id, nlen), path}, - {LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0), NULL})); + {LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0)})); if (err) { err = LFS_ERR_NAMETOOLONG; goto cleanup; @@ -3145,7 +3133,7 @@ int lfs_remove(lfs_t *lfs, const char *path) { // delete the entry err = lfs_dir_commit(lfs, &cwd, LFS_MKATTRS( - {LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(tag), 0), NULL})); + {LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(tag), 0)})); if (err) { LFS_TRACE("lfs_remove -> %d", err); return err; @@ -3186,7 +3174,8 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { lfs_mdir_t oldcwd; lfs_stag_t oldtag = lfs_dir_find(lfs, &oldcwd, &oldpath, NULL); if (oldtag < 0 || lfs_tag_id(oldtag) == 0x3ff) { - LFS_TRACE("lfs_rename -> %"PRId32, (oldtag < 0) ? oldtag : LFS_ERR_INVAL); + LFS_TRACE("lfs_rename -> %"PRId32, + (oldtag < 0) ? oldtag : LFS_ERR_INVAL); return (oldtag < 0) ? (int)oldtag : LFS_ERR_INVAL; } @@ -3196,7 +3185,8 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { lfs_stag_t prevtag = lfs_dir_find(lfs, &newcwd, &newpath, &newid); if ((prevtag < 0 || lfs_tag_id(prevtag) == 0x3ff) && !(prevtag == LFS_ERR_NOENT && newid != 0x3ff)) { - LFS_TRACE("lfs_rename -> %"PRId32, (prevtag < 0) ? prevtag : LFS_ERR_INVAL); + LFS_TRACE("lfs_rename -> %"PRId32, + (prevtag < 0) ? prevtag : LFS_ERR_INVAL); return (prevtag < 0) ? (int)prevtag : LFS_ERR_INVAL; } @@ -3239,26 +3229,28 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { } // create move to fix later + bool samepair = (lfs_pair_cmp(oldcwd.pair, newcwd.pair) == 0); uint16_t newoldtagid = lfs_tag_id(oldtag); - if (lfs_pair_cmp(oldcwd.pair, newcwd.pair) == 0 && - prevtag == LFS_ERR_NOENT && newid <= newoldtagid) { - // there is a small chance we are being renamed in the same directory - // to an id less than our old id, the global update to handle this - // is a bit messy - newoldtagid += 1; + if (samepair) { + // there is a small chance we are being renamed in the same + // directory/ to an id less than our old id, the global update + // to handle this is a bit messy + if (prevtag == LFS_ERR_NOENT && newid <= newoldtagid) { + newoldtagid += 1; + } + } else { + lfs_fs_prepmove(lfs, newoldtagid, oldcwd.pair); } - lfs_fs_prepmove(lfs, newoldtagid, oldcwd.pair); - // move over all attributes err = lfs_dir_commit(lfs, &newcwd, LFS_MKATTRS( - {prevtag != LFS_ERR_NOENT - ? LFS_MKTAG(LFS_TYPE_DELETE, newid, 0) - : LFS_MKTAG(LFS_FROM_NOOP, 0, 0), NULL}, - {LFS_MKTAG(LFS_TYPE_CREATE, newid, 0), NULL}, - {LFS_MKTAG(lfs_tag_type3(oldtag), newid, strlen(newpath)), - newpath}, - {LFS_MKTAG(LFS_FROM_MOVE, newid, lfs_tag_id(oldtag)), &oldcwd})); + {LFS_MKTAG_IF(prevtag != LFS_ERR_NOENT, + LFS_TYPE_DELETE, newid, 0)}, + {LFS_MKTAG(LFS_TYPE_CREATE, newid, 0)}, + {LFS_MKTAG(lfs_tag_type3(oldtag), newid, strlen(newpath)), newpath}, + {LFS_MKTAG(LFS_FROM_MOVE, newid, lfs_tag_id(oldtag)), &oldcwd}, + {LFS_MKTAG_IF(samepair, + LFS_TYPE_DELETE, newoldtagid, 0)})); if (err) { LFS_TRACE("lfs_rename -> %d", err); return err; @@ -3266,8 +3258,11 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { // let commit clean up after move (if we're different! otherwise move // logic already fixed it for us) - if (lfs_pair_cmp(oldcwd.pair, newcwd.pair) != 0) { - err = lfs_dir_commit(lfs, &oldcwd, NULL, 0); + if (!samepair && lfs_gstate_hasmove(&lfs->gstate)) { + // prep gstate and delete move id + lfs_fs_prepmove(lfs, 0x3ff, NULL); + err = lfs_dir_commit(lfs, &oldcwd, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(oldtag), 0)})); if (err) { LFS_TRACE("lfs_rename -> %d", err); return err; @@ -3557,7 +3552,7 @@ int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) { lfs_superblock_tole32(&superblock); err = lfs_dir_commit(lfs, &root, LFS_MKATTRS( - {LFS_MKTAG(LFS_TYPE_CREATE, 0, 0), NULL}, + {LFS_MKTAG(LFS_TYPE_CREATE, 0, 0)}, {LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8), "littlefs"}, {LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)), &superblock})); @@ -3922,8 +3917,25 @@ static int lfs_fs_relocate(lfs_t *lfs, // update disk, this creates a desync lfs_fs_preporphans(lfs, +1); + // fix pending move in this pair? this looks like an optimization but + // is in fact _required_ since relocating may outdate the move. + uint16_t moveid = 0x3ff; + if (lfs_gstate_hasmovehere(&lfs->gpending, parent.pair)) { + moveid = lfs_tag_id(lfs->gpending.tag); + LFS_DEBUG("Fixing move while relocating " + "%"PRIx32" %"PRIx32" %"PRIx16"\n", + parent.pair[0], parent.pair[1], moveid); + lfs_fs_prepmove(lfs, 0x3ff, NULL); + if (moveid < lfs_tag_id(tag)) { + tag -= LFS_MKTAG(0, 1, 0); + } + } + lfs_pair_tole32(newpair); - int err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS({tag, newpair})); + int err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS( + {LFS_MKTAG_IF(moveid != 0x3ff, + LFS_TYPE_DELETE, moveid, 0)}, + {tag, newpair})); lfs_pair_fromle32(newpair); if (err) { return err; @@ -3941,9 +3953,22 @@ static int lfs_fs_relocate(lfs_t *lfs, // if we can't find dir, it must be new if (err != LFS_ERR_NOENT) { + // fix pending move in this pair? this looks like an optimization but + // is in fact _required_ since relocating may outdate the move. + uint16_t moveid = 0x3ff; + if (lfs_gstate_hasmovehere(&lfs->gpending, parent.pair)) { + moveid = lfs_tag_id(lfs->gpending.tag); + LFS_DEBUG("Fixing move while relocating " + "%"PRIx32" %"PRIx32" %"PRIx16"\n", + parent.pair[0], parent.pair[1], moveid); + lfs_fs_prepmove(lfs, 0x3ff, NULL); + } + // replace bad pair, either we clean up desync, or no desync occured lfs_pair_tole32(newpair); err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS( + {LFS_MKTAG_IF(moveid != 0x3ff, + LFS_TYPE_DELETE, moveid, 0)}, {LFS_MKTAG(LFS_TYPE_TAIL + parent.split, 0x3ff, 8), newpair})); lfs_pair_fromle32(newpair); if (err) { @@ -3987,8 +4012,11 @@ static int lfs_fs_demove(lfs_t *lfs) { return err; } - // rely on cancel logic inside commit - err = lfs_dir_commit(lfs, &movedir, NULL, 0); + // prep gstate and delete move id + uint16_t moveid = lfs_tag_id(lfs->gstate.tag); + lfs_fs_prepmove(lfs, 0x3ff, NULL); + err = lfs_dir_commit(lfs, &movedir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_DELETE, moveid, 0)})); if (err) { return err; } @@ -4630,13 +4658,15 @@ int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) { lfs1_entry_tole32(&entry1.d); err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS( - {LFS_MKTAG(LFS_TYPE_CREATE, id, 0), NULL}, - {LFS_MKTAG( - isdir ? LFS_TYPE_DIR : LFS_TYPE_REG, - id, entry1.d.nlen), name}, - {LFS_MKTAG( - isdir ? LFS_TYPE_DIRSTRUCT : LFS_TYPE_CTZSTRUCT, - id, sizeof(entry1.d.u)), &entry1.d.u})); + {LFS_MKTAG(LFS_TYPE_CREATE, id, 0)}, + {LFS_MKTAG_IF_ELSE(isdir, + LFS_TYPE_DIR, id, entry1.d.nlen, + LFS_TYPE_REG, id, entry1.d.nlen), + name}, + {LFS_MKTAG_IF_ELSE(isdir, + LFS_TYPE_DIRSTRUCT, id, sizeof(entry1.d.u), + LFS_TYPE_CTZSTRUCT, id, sizeof(entry1.d.u)), + &entry1.d.u})); lfs1_entry_fromle32(&entry1.d); if (err) { goto cleanup; @@ -4659,8 +4689,7 @@ int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) { lfs_pair_tole32(dir2.pair); err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS( - {LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), - dir1.d.tail})); + {LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir1.d.tail})); lfs_pair_fromle32(dir2.pair); if (err) { goto cleanup; @@ -4733,7 +4762,7 @@ int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) { lfs_superblock_tole32(&superblock); err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS( - {LFS_MKTAG(LFS_TYPE_CREATE, 0, 0), NULL}, + {LFS_MKTAG(LFS_TYPE_CREATE, 0, 0)}, {LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8), "littlefs"}, {LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)), &superblock})); diff --git a/scripts/readmdir.py b/scripts/readmdir.py index a60acbe7..1730f029 100755 --- a/scripts/readmdir.py +++ b/scripts/readmdir.py @@ -106,6 +106,13 @@ def mkmask(self): 0x3ff if self.isattr else 0, 0) + def chid(self, nid): + ntag = Tag(self.type, nid, self.size) + if hasattr(self, 'off'): ntag.off = self.off + if hasattr(self, 'data'): ntag.data = self.data + if hasattr(self, 'crc'): ntag.crc = self.crc + return ntag + def typerepr(self): if self.is_('crc') and getattr(self, 'crc', 0xffffffff) != 0xffffffff: return 'crc (bad)' @@ -197,22 +204,29 @@ def __init__(self, blocks): crc = 0 tag = Tag(int(tag) ^ ((tag.type & 1) << 31)) + # find active ids + self.ids = list(it.takewhile( + lambda id: Tag('name', id, 0) in self, + it.count())) + # find most recent tags self.tags = [] for tag in self.log: if tag.is_('crc') or tag.is_('splice'): continue - - if tag in self and self[tag] is tag: - self.tags.append(tag) + elif tag.id == 0x3ff: + if tag in self and self[tag] is tag: + self.tags.append(tag) + else: + # id could have change, I know this is messy and slow + # but it works + for id in self.ids: + ntag = tag.chid(id) + if ntag in self and self[ntag] is tag: + self.tags.append(ntag) self.tags = sorted(self.tags) - # and ids - self.ids = list(it.takewhile( - lambda id: Tag('name', id, 0) in self, - it.count())) - def __bool__(self): return bool(self.log) @@ -247,8 +261,8 @@ def __getitem__(self, args): gdiff += tag.schunk - if (int(gmask) & int(tag)) == (int(gmask) & int( - Tag(gtag.type, gtag.id - gdiff, gtag.size))): + if ((int(gmask) & int(tag)) == + (int(gmask) & int(gtag.chid(gtag.id - gdiff)))): if tag.size == 0x3ff: # deleted break diff --git a/scripts/readtree.py b/scripts/readtree.py index 2bae10e0..7d112fd9 100755 --- a/scripts/readtree.py +++ b/scripts/readtree.py @@ -38,6 +38,8 @@ def dumptags(args, mdir, f): ' 0; } else if (lfs_stat(&lfs, "d/hello", &info) == 0) { // success + lfs_unmount(&lfs) => 0; break; } else { // create file @@ -807,7 +808,7 @@ code = ''' if (lfs_stat(&lfs, "b/hi", &info) == 0) { assert(strcmp(info.name, "hi") == 0); assert(info.type == LFS_TYPE_DIR); - count += 1; + count += 1; } if (lfs_stat(&lfs, "c/hi", &info) == 0) { assert(strcmp(info.name, "hi") == 0); @@ -830,6 +831,7 @@ code = ''' } else if (lfs_stat(&lfs, "c/hi", &info) == 0) { lfs_rename(&lfs, "c/hi", "d/hi") => 0; } else if (lfs_stat(&lfs, "d/hi", &info) == 0) { + lfs_unmount(&lfs) => 0; break; // success } else { // create dir and rename for atomicity @@ -955,3 +957,839 @@ code = ''' lfs_file_close(&lfs, &file) => 0; lfs_unmount(&lfs) => 0; ''' + +# Other specific corner cases +[[case]] # create + delete in same commit with neighbors +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // littlefs keeps files sorted, so we know the order these will be in + lfs_file_open(&lfs, &file, "/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.1", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/2.in_between", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.2", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/4.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.3", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_t files[3]; + lfs_file_open(&lfs, &files[0], "0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "2.in_between", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "4.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.4", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.5", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.6", 7) => 7; + + // rename file while everything is open, this triggers both + // a create and delete simultaneously + lfs_rename(&lfs, "/1.move_me", "/3.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + + // check that nothing was corrupted + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.in_between") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "3.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "4.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.4") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/2.in_between", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.5") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/4.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.6") == 0); + lfs_file_close(&lfs, &file) => 0; + + // now move back + lfs_file_open(&lfs, &files[0], "0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "2.in_between", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "4.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.7", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.8", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.9", 7) => 7; + + // rename file while everything is open, this triggers both + // a create and delete simultaneously + lfs_rename(&lfs, "/3.move_me", "/1.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + + // and check that nothing was corrupted again + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "1.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.in_between") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "4.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.7") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/2.in_between", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.8") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/4.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.9") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +# Other specific corner cases +[[case]] # create + delete + delete in same commit with neighbors +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // littlefs keeps files sorted, so we know the order these will be in + lfs_file_open(&lfs, &file, "/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/3.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "remove me", + sizeof("remove me")) => sizeof("remove me"); + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.1", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/2.in_between", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.2", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/4.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.3", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_t files[3]; + lfs_file_open(&lfs, &files[0], "0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "2.in_between", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "4.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.4", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.5", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.6", 7) => 7; + + // rename file while everything is open, this triggers both + // a create and delete simultaneously + lfs_rename(&lfs, "/1.move_me", "/3.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + + // check that nothing was corrupted + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.in_between") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "3.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "4.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.4") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/2.in_between", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.5") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/4.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.6") == 0); + lfs_file_close(&lfs, &file) => 0; + + // now move back + lfs_file_open(&lfs, &file, "/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "remove me", + sizeof("remove me")) => sizeof("remove me"); + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &files[0], "0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "2.in_between", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "4.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.7", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.8", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.9", 7) => 7; + + // rename file while everything is open, this triggers both + // a create and delete simultaneously + lfs_rename(&lfs, "/3.move_me", "/1.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + + // and check that nothing was corrupted again + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "1.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.in_between") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "4.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.7") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/2.in_between", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.8") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/4.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.9") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # create + delete in different dirs with neighbors +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + // littlefs keeps files sorted, so we know the order these will be in + lfs_mkdir(&lfs, "/dir.1") => 0; + lfs_mkdir(&lfs, "/dir.2") => 0; + lfs_file_open(&lfs, &file, "/dir.1/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.2/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "remove me", + sizeof("remove me")) => sizeof("remove me"); + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/dir.1/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.1", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.1/2.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.2", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/dir.2/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.3", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.2/2.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.4", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_t files[4]; + lfs_file_open(&lfs, &files[0], "/dir.1/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "/dir.1/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "/dir.2/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[3], "/dir.2/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.5", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.6", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.7", 7) => 7; + lfs_file_write(&lfs, &files[3], "test.8", 7) => 7; + + // rename file while everything is open, this triggers both + // a create and delete as it overwrites the destination file + lfs_rename(&lfs, "/dir.1/1.move_me", "/dir.2/1.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + lfs_file_close(&lfs, &files[3]) => 0; + + // check that nothing was corrupted + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "dir.1") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "dir.2") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/dir.1") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/dir.2") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "1.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/dir.1/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.5") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.1/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.6") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.2/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.7") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.2/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.8") == 0); + lfs_file_close(&lfs, &file) => 0; + + // now move back + lfs_file_open(&lfs, &file, "/dir.1/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "remove me", + sizeof("remove me")) => sizeof("remove me"); + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &files[0], "/dir.1/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "/dir.1/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "/dir.2/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[3], "/dir.2/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.9", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.a", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.b", 7) => 7; + lfs_file_write(&lfs, &files[3], "test.c", 7) => 7; + + // rename file while everything is open, this triggers both + // a create and delete simultaneously + lfs_rename(&lfs, "/dir.2/1.move_me", "/dir.1/1.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + lfs_file_close(&lfs, &files[3]) => 0; + + // and check that nothing was corrupted again + lfs_dir_open(&lfs, &dir, "/") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "dir.1") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "dir.2") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/dir.1") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "1.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 0); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/dir.2") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/dir.1/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.9") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.1/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.a") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.2/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.b") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/dir.2/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.c") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move fix in relocation +in = "lfs.c" +define.RELOCATIONS = 'range(0x3+1)' +define.LFS_ERASE_CYCLES = 0xffffffff +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + lfs_mkdir(&lfs, "/parent") => 0; + lfs_mkdir(&lfs, "/parent/child") => 0; + + lfs_file_open(&lfs, &file, "/parent/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "move me", + sizeof("move me")) => sizeof("move me"); + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/parent/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.1", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/2.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.2", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.3", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/2.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.4", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_t files[4]; + lfs_file_open(&lfs, &files[0], "/parent/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "/parent/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "/parent/child/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[3], "/parent/child/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.5", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.6", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.7", 7) => 7; + lfs_file_write(&lfs, &files[3], "test.8", 7) => 7; + + // force specific directories to relocate + if (RELOCATIONS & 0x1) { + lfs_dir_open(&lfs, &dir, "/parent"); + lfs_testbd_setwear(&cfg, dir.m.pair[0], 0xffffffff) => 0; + lfs_testbd_setwear(&cfg, dir.m.pair[1], 0xffffffff) => 0; + lfs_dir_close(&lfs, &dir) => 0; + } + if (RELOCATIONS & 0x2) { + lfs_dir_open(&lfs, &dir, "/parent/child"); + lfs_testbd_setwear(&cfg, dir.m.pair[0], 0xffffffff) => 0; + lfs_testbd_setwear(&cfg, dir.m.pair[1], 0xffffffff) => 0; + lfs_dir_close(&lfs, &dir) => 0; + } + + // ok, now we move the file, this creates a move that needs to be + // fixed, possibly in a metadata-pair that needs to be relocated + // + // the worst case is if we need to relocate and we need to implicit + // fix the move in our parent before it falls out of date + lfs_rename(&lfs, "/parent/1.move_me", "/parent/child/1.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + lfs_file_close(&lfs, &files[3]) => 0; + + // check that nothing was corrupted + lfs_dir_open(&lfs, &dir, "/parent") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "child") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/parent/child") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "1.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == sizeof("move me")); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/parent/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.5") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.6") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.7") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.8") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # move fix in relocation with predecessor +in = "lfs.c" +define.RELOCATIONS = 'range(0x7+1)' +define.LFS_ERASE_CYCLES = 0xffffffff +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + + lfs_mkdir(&lfs, "/parent") => 0; + lfs_mkdir(&lfs, "/parent/child") => 0; + lfs_mkdir(&lfs, "/parent/sibling") => 0; + + lfs_file_open(&lfs, &file, "/parent/sibling/1.move_me", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "move me", + sizeof("move me")) => sizeof("move me"); + lfs_file_close(&lfs, &file) => 0; + + lfs_file_open(&lfs, &file, "/parent/sibling/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.1", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/sibling/2.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.2", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/0.before", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.3", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/2.after", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_write(&lfs, &file, "test.4", 7) => 7; + lfs_file_close(&lfs, &file) => 0; + + lfs_file_t files[4]; + lfs_file_open(&lfs, &files[0], "/parent/sibling/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[1], "/parent/sibling/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[2], "/parent/child/0.before", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_open(&lfs, &files[3], "/parent/child/2.after", + LFS_O_WRONLY | LFS_O_TRUNC) => 0; + lfs_file_write(&lfs, &files[0], "test.5", 7) => 7; + lfs_file_write(&lfs, &files[1], "test.6", 7) => 7; + lfs_file_write(&lfs, &files[2], "test.7", 7) => 7; + lfs_file_write(&lfs, &files[3], "test.8", 7) => 7; + + // force specific directories to relocate + if (RELOCATIONS & 0x1) { + lfs_dir_open(&lfs, &dir, "/parent"); + lfs_testbd_setwear(&cfg, dir.m.pair[0], 0xffffffff) => 0; + lfs_testbd_setwear(&cfg, dir.m.pair[1], 0xffffffff) => 0; + lfs_dir_close(&lfs, &dir) => 0; + } + if (RELOCATIONS & 0x2) { + lfs_dir_open(&lfs, &dir, "/parent/sibling"); + lfs_testbd_setwear(&cfg, dir.m.pair[0], 0xffffffff) => 0; + lfs_testbd_setwear(&cfg, dir.m.pair[1], 0xffffffff) => 0; + lfs_dir_close(&lfs, &dir) => 0; + } + if (RELOCATIONS & 0x4) { + lfs_dir_open(&lfs, &dir, "/parent/child"); + lfs_testbd_setwear(&cfg, dir.m.pair[0], 0xffffffff) => 0; + lfs_testbd_setwear(&cfg, dir.m.pair[1], 0xffffffff) => 0; + lfs_dir_close(&lfs, &dir) => 0; + } + + // ok, now we move the file, this creates a move that needs to be + // fixed, possibly in a metadata-pair that needs to be relocated + // + // and now relocations can force us to need to fix our move in either + // the parent or child before things break + lfs_rename(&lfs, + "/parent/sibling/1.move_me", + "/parent/child/1.move_me") => 0; + + lfs_file_close(&lfs, &files[0]) => 0; + lfs_file_close(&lfs, &files[1]) => 0; + lfs_file_close(&lfs, &files[2]) => 0; + lfs_file_close(&lfs, &files[3]) => 0; + + // check that nothing was corrupted + lfs_dir_open(&lfs, &dir, "/parent") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "child") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "sibling") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/parent/sibling") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_dir_open(&lfs, &dir, "/parent/child") => 0; + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "0.before") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "1.move_me") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == sizeof("move me")); + lfs_dir_read(&lfs, &dir, &info) => 1; + assert(strcmp(info.name, "2.after") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == 7); + lfs_dir_read(&lfs, &dir, &info) => 0; + lfs_dir_close(&lfs, &dir) => 0; + + lfs_file_open(&lfs, &file, "/parent/sibling/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.5") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/sibling/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.6") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/0.before", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.7") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_file_open(&lfs, &file, "/parent/child/2.after", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, 7) => 7; + assert(strcmp((char*)buffer, "test.8") == 0); + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' From a5d614fbfbf19b8605e08c28a53bc69ea3179a3e Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 21 Jan 2020 22:18:19 -0600 Subject: [PATCH 12/41] Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer. --- lfs.c | 279 ++++++++++-------- lfs.h | 12 +- scripts/readtree.py | 19 +- tests_/test_move.toml | 16 + .../{test_orphan.toml => test_orphans.toml} | 61 ++++ tests_/test_relocations.toml | 160 +++++++++- 6 files changed, 412 insertions(+), 135 deletions(-) rename tests_/{test_orphan.toml => test_orphans.toml} (50%) diff --git a/lfs.c b/lfs.c index 6a0a9245..bc6e1bc6 100644 --- a/lfs.c +++ b/lfs.c @@ -323,14 +323,13 @@ struct lfs_diskoff { sizeof((struct lfs_mattr[]){__VA_ARGS__}) / sizeof(struct lfs_mattr) // operations on global state -static inline void lfs_gstate_xor(struct lfs_gstate *a, - const struct lfs_gstate *b) { +static inline void lfs_gstate_xor(lfs_gstate_t *a, const lfs_gstate_t *b) { for (int i = 0; i < 3; i++) { ((uint32_t*)a)[i] ^= ((const uint32_t*)b)[i]; } } -static inline bool lfs_gstate_iszero(const struct lfs_gstate *a) { +static inline bool lfs_gstate_iszero(const lfs_gstate_t *a) { for (int i = 0; i < 3; i++) { if (((uint32_t*)a)[i] != 0) { return false; @@ -339,43 +338,30 @@ static inline bool lfs_gstate_iszero(const struct lfs_gstate *a) { return true; } -static inline bool lfs_gstate_hasorphans(const struct lfs_gstate *a) { +static inline bool lfs_gstate_hasorphans(const lfs_gstate_t *a) { return lfs_tag_size(a->tag); } -static inline uint8_t lfs_gstate_getorphans(const struct lfs_gstate *a) { +static inline uint8_t lfs_gstate_getorphans(const lfs_gstate_t *a) { return lfs_tag_size(a->tag); } -static inline bool lfs_gstate_hasmove(const struct lfs_gstate *a) { +static inline bool lfs_gstate_hasmove(const lfs_gstate_t *a) { return lfs_tag_type1(a->tag); } -static inline bool lfs_gstate_hasmovehere(const struct lfs_gstate *a, +static inline bool lfs_gstate_hasmovehere(const lfs_gstate_t *a, const lfs_block_t *pair) { return lfs_tag_type1(a->tag) && lfs_pair_cmp(a->pair, pair) == 0; } -static inline void lfs_gstate_xororphans(struct lfs_gstate *a, - const struct lfs_gstate *b, bool orphans) { - a->tag ^= LFS_MKTAG(0x800, 0, 0) & (b->tag ^ ((uint32_t)orphans << 31)); -} - -static inline void lfs_gstate_xormove(struct lfs_gstate *a, - const struct lfs_gstate *b, uint16_t id, const lfs_block_t pair[2]) { - a->tag ^= LFS_MKTAG(0x7ff, 0x3ff, 0) & (b->tag ^ ( - (id != 0x3ff) ? LFS_MKTAG(LFS_TYPE_DELETE, id, 0) : 0)); - a->pair[0] ^= b->pair[0] ^ ((id != 0x3ff) ? pair[0] : 0); - a->pair[1] ^= b->pair[1] ^ ((id != 0x3ff) ? pair[1] : 0); -} - -static inline void lfs_gstate_fromle32(struct lfs_gstate *a) { +static inline void lfs_gstate_fromle32(lfs_gstate_t *a) { a->tag = lfs_fromle32(a->tag); a->pair[0] = lfs_fromle32(a->pair[0]); a->pair[1] = lfs_fromle32(a->pair[1]); } -static inline void lfs_gstate_tole32(struct lfs_gstate *a) { +static inline void lfs_gstate_tole32(lfs_gstate_t *a) { a->tag = lfs_tole32(a->tag); a->pair[0] = lfs_tole32(a->pair[0]); a->pair[1] = lfs_tole32(a->pair[1]); @@ -506,8 +492,9 @@ static lfs_stag_t lfs_dir_getslice(lfs_t *lfs, const lfs_mdir_t *dir, lfs_tag_t ntag = dir->etag; lfs_stag_t gdiff = 0; - if (lfs_gstate_hasmovehere(&lfs->gstate, dir->pair) && - lfs_tag_id(gtag) <= lfs_tag_id(lfs->gstate.tag)) { + if (lfs_gstate_hasmovehere(&lfs->gdisk, dir->pair) && + lfs_tag_id(gmask) != 0 && + lfs_tag_id(lfs->gdisk.tag) <= lfs_tag_id(gtag)) { // synthetic moves gdiff -= LFS_MKTAG(0, 1, 0); } @@ -944,11 +931,11 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, // consider what we have good enough if (dir->off > 0) { // synthetic move - if (lfs_gstate_hasmovehere(&lfs->gstate, dir->pair)) { - if (lfs_tag_id(lfs->gstate.tag) == lfs_tag_id(besttag)) { + if (lfs_gstate_hasmovehere(&lfs->gdisk, dir->pair)) { + if (lfs_tag_id(lfs->gdisk.tag) == lfs_tag_id(besttag)) { besttag |= 0x80000000; } else if (besttag != -1 && - lfs_tag_id(lfs->gstate.tag) < lfs_tag_id(besttag)) { + lfs_tag_id(lfs->gdisk.tag) < lfs_tag_id(besttag)) { besttag -= LFS_MKTAG(0, 1, 0); } } @@ -986,8 +973,8 @@ static int lfs_dir_fetch(lfs_t *lfs, } static int lfs_dir_getgstate(lfs_t *lfs, const lfs_mdir_t *dir, - struct lfs_gstate *gstate) { - struct lfs_gstate temp; + lfs_gstate_t *gstate) { + lfs_gstate_t temp; lfs_stag_t res = lfs_dir_get(lfs, dir, LFS_MKTAG(0x7ff, 0, 0), LFS_MKTAG(LFS_TYPE_MOVESTATE, 0, sizeof(temp)), &temp); if (res < 0 && res != LFS_ERR_NOENT) { @@ -1527,13 +1514,6 @@ static int lfs_dir_compact(lfs_t *lfs, // begin loop to commit compaction to blocks until a compact sticks while (true) { { - // There's nothing special about our global delta, so feed it into - // our local global delta - int err = lfs_dir_getgstate(lfs, dir, &lfs->gdelta); - if (err) { - return err; - } - // setup commit state struct lfs_commit commit = { .block = dir->pair[1], @@ -1546,7 +1526,7 @@ static int lfs_dir_compact(lfs_t *lfs, }; // erase block to write to - err = lfs_bd_erase(lfs, dir->pair[1]); + int err = lfs_bd_erase(lfs, dir->pair[1]); if (err) { if (err == LFS_ERR_CORRUPT) { goto relocate; @@ -1596,14 +1576,25 @@ static int lfs_dir_compact(lfs_t *lfs, } } - if (!relocated && !lfs_gstate_iszero(&lfs->gdelta)) { - // commit any globals, unless we're relocating, - // in which case our parent will steal our globals - lfs_gstate_tole32(&lfs->gdelta); + // bring over gstate? + lfs_gstate_t delta = {0}; + if (!relocated) { + lfs_gstate_xor(&delta, &lfs->gdisk); + lfs_gstate_xor(&delta, &lfs->gstate); + } + lfs_gstate_xor(&delta, &lfs->gdelta); + delta.tag &= ~LFS_MKTAG(0, 0, 0x3ff); + + err = lfs_dir_getgstate(lfs, dir, &delta); + if (err) { + return err; + } + + if (!lfs_gstate_iszero(&delta)) { + lfs_gstate_tole32(&delta); err = lfs_dir_commitattr(lfs, &commit, LFS_MKTAG(LFS_TYPE_MOVESTATE, 0x3ff, - sizeof(lfs->gdelta)), &lfs->gdelta); - lfs_gstate_fromle32(&lfs->gdelta); + sizeof(delta)), &delta); if (err) { if (err == LFS_ERR_CORRUPT) { goto relocate; @@ -1612,6 +1603,7 @@ static int lfs_dir_compact(lfs_t *lfs, } } + // complete commit with crc err = lfs_dir_commitcrc(lfs, &commit); if (err) { if (err == LFS_ERR_CORRUPT) { @@ -1626,6 +1618,11 @@ static int lfs_dir_compact(lfs_t *lfs, dir->count = end - begin; dir->off = commit.off; dir->etag = commit.ptag; + // update gstate + lfs->gdelta = (lfs_gstate_t){0}; + if (!relocated) { + lfs->gdisk = lfs->gstate; + } } break; @@ -1653,10 +1650,7 @@ static int lfs_dir_compact(lfs_t *lfs, continue; } - if (!relocated) { - lfs->gstate = lfs->gpending; - lfs->gdelta = (struct lfs_gstate){0}; - } else { + if (relocated) { // update references if we relocated LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); @@ -1747,17 +1741,21 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } // commit any global diffs if we have any - if (!lfs_gstate_iszero(&lfs->gdelta)) { - err = lfs_dir_getgstate(lfs, dir, &lfs->gdelta); + lfs_gstate_t delta = {0}; + lfs_gstate_xor(&delta, &lfs->gstate); + lfs_gstate_xor(&delta, &lfs->gdisk); + lfs_gstate_xor(&delta, &lfs->gdelta); + delta.tag &= ~LFS_MKTAG(0, 0, 0x3ff); + if (!lfs_gstate_iszero(&delta)) { + err = lfs_dir_getgstate(lfs, dir, &delta); if (err) { return err; } - lfs_gstate_tole32(&lfs->gdelta); + lfs_gstate_tole32(&delta); err = lfs_dir_commitattr(lfs, &commit, LFS_MKTAG(LFS_TYPE_MOVESTATE, 0x3ff, - sizeof(lfs->gdelta)), &lfs->gdelta); - lfs_gstate_fromle32(&lfs->gdelta); + sizeof(delta)), &delta); if (err) { if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) { goto compact; @@ -1780,8 +1778,8 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, dir->off = commit.off; dir->etag = commit.ptag; // and update gstate - lfs->gstate = lfs->gpending; - lfs->gdelta = (struct lfs_gstate){0}; + lfs->gdisk = lfs->gstate; + lfs->gdelta = (lfs_gstate_t){0}; } else { compact: // fall back to compaction @@ -1855,9 +1853,10 @@ int lfs_mkdir(lfs_t *lfs, const char *path) { return err; } - lfs_mdir_t cwd; + struct lfs_mlist cwd; + cwd.next = lfs->mlist; uint16_t id; - err = lfs_dir_find(lfs, &cwd, &path, &id); + err = lfs_dir_find(lfs, &cwd.m, &path, &id); if (!(err == LFS_ERR_NOENT && id != 0x3ff)) { LFS_TRACE("lfs_mkdir -> %d", (err < 0) ? err : LFS_ERR_EXIST); return (err < 0) ? err : LFS_ERR_EXIST; @@ -1880,7 +1879,7 @@ int lfs_mkdir(lfs_t *lfs, const char *path) { } // find end of list - lfs_mdir_t pred = cwd; + lfs_mdir_t pred = cwd.m; while (pred.split) { err = lfs_dir_fetch(lfs, &pred, pred.tail); if (err) { @@ -1900,27 +1899,39 @@ int lfs_mkdir(lfs_t *lfs, const char *path) { } // current block end of list? - if (cwd.split) { + if (cwd.m.split) { // update tails, this creates a desync lfs_fs_preporphans(lfs, +1); + + // it's possible our predecessor has to be relocated, and if + // our parent is our predecessor's predecessor, this could have + // caused our parent to go out of date, fortunately we can hook + // ourselves into littlefs to catch this + cwd.type = 0; + cwd.id = 0; + lfs->mlist = &cwd; + lfs_pair_tole32(dir.pair); err = lfs_dir_commit(lfs, &pred, LFS_MKATTRS( {LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir.pair})); lfs_pair_fromle32(dir.pair); if (err) { + lfs->mlist = cwd.next; LFS_TRACE("lfs_mkdir -> %d", err); return err; } + + lfs->mlist = cwd.next; lfs_fs_preporphans(lfs, -1); } // now insert into our parent block lfs_pair_tole32(dir.pair); - err = lfs_dir_commit(lfs, &cwd, LFS_MKATTRS( + err = lfs_dir_commit(lfs, &cwd.m, LFS_MKATTRS( {LFS_MKTAG(LFS_TYPE_CREATE, id, 0), NULL}, {LFS_MKTAG(LFS_TYPE_DIR, id, nlen), path}, {LFS_MKTAG(LFS_TYPE_DIRSTRUCT, id, 8), dir.pair}, - {LFS_MKTAG_IF(!cwd.split, + {LFS_MKTAG_IF(!cwd.m.split, LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir.pair})); lfs_pair_fromle32(dir.pair); if (err) { @@ -3104,7 +3115,8 @@ int lfs_remove(lfs_t *lfs, const char *path) { return (tag < 0) ? (int)tag : LFS_ERR_INVAL; } - lfs_mdir_t dir; + struct lfs_mlist dir; + dir.next = lfs->mlist; if (lfs_tag_type3(tag) == LFS_TYPE_DIR) { // must be empty before removal lfs_block_t pair[2]; @@ -3116,40 +3128,48 @@ int lfs_remove(lfs_t *lfs, const char *path) { } lfs_pair_fromle32(pair); - err = lfs_dir_fetch(lfs, &dir, pair); + err = lfs_dir_fetch(lfs, &dir.m, pair); if (err) { LFS_TRACE("lfs_remove -> %d", err); return err; } - if (dir.count > 0 || dir.split) { + if (dir.m.count > 0 || dir.m.split) { LFS_TRACE("lfs_remove -> %d", LFS_ERR_NOTEMPTY); return LFS_ERR_NOTEMPTY; } // mark fs as orphaned lfs_fs_preporphans(lfs, +1); + + // I know it's crazy but yes, dir can be changed by our parent's + // commit (if predecessor is child) + dir.type = 0; + dir.id = 0; + lfs->mlist = &dir; } // delete the entry err = lfs_dir_commit(lfs, &cwd, LFS_MKATTRS( {LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(tag), 0)})); if (err) { + lfs->mlist = dir.next; LFS_TRACE("lfs_remove -> %d", err); return err; } + lfs->mlist = dir.next; if (lfs_tag_type3(tag) == LFS_TYPE_DIR) { // fix orphan lfs_fs_preporphans(lfs, -1); - err = lfs_fs_pred(lfs, dir.pair, &cwd); + err = lfs_fs_pred(lfs, dir.m.pair, &cwd); if (err) { LFS_TRACE("lfs_remove -> %d", err); return err; } - err = lfs_dir_drop(lfs, &cwd, &dir); + err = lfs_dir_drop(lfs, &cwd, &dir.m); if (err) { LFS_TRACE("lfs_remove -> %d", err); return err; @@ -3190,7 +3210,12 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { return (prevtag < 0) ? (int)prevtag : LFS_ERR_INVAL; } - lfs_mdir_t prevdir; + // if we're in the same pair there's a few special cases... + bool samepair = (lfs_pair_cmp(oldcwd.pair, newcwd.pair) == 0); + uint16_t newoldid = lfs_tag_id(oldtag); + + struct lfs_mlist prevdir; + prevdir.next = lfs->mlist; if (prevtag == LFS_ERR_NOENT) { // check that name fits lfs_size_t nlen = strlen(newpath); @@ -3198,9 +3223,20 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { LFS_TRACE("lfs_rename -> %d", LFS_ERR_NAMETOOLONG); return LFS_ERR_NAMETOOLONG; } + + // there is a small chance we are being renamed in the same + // directory/ to an id less than our old id, the global update + // to handle this is a bit messy + if (samepair && newid <= newoldid) { + newoldid += 1; + } } else if (lfs_tag_type3(prevtag) != lfs_tag_type3(oldtag)) { LFS_TRACE("lfs_rename -> %d", LFS_ERR_ISDIR); return LFS_ERR_ISDIR; + } else if (samepair && newid == newoldid) { + // we're renaming to ourselves?? + LFS_TRACE("lfs_rename -> %d", 0); + return 0; } else if (lfs_tag_type3(prevtag) == LFS_TYPE_DIR) { // must be empty before removal lfs_block_t prevpair[2]; @@ -3213,33 +3249,29 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { lfs_pair_fromle32(prevpair); // must be empty before removal - err = lfs_dir_fetch(lfs, &prevdir, prevpair); + err = lfs_dir_fetch(lfs, &prevdir.m, prevpair); if (err) { LFS_TRACE("lfs_rename -> %d", err); return err; } - if (prevdir.count > 0 || prevdir.split) { + if (prevdir.m.count > 0 || prevdir.m.split) { LFS_TRACE("lfs_rename -> %d", LFS_ERR_NOTEMPTY); return LFS_ERR_NOTEMPTY; } // mark fs as orphaned lfs_fs_preporphans(lfs, +1); + + // I know it's crazy but yes, dir can be changed by our parent's + // commit (if predecessor is child) + prevdir.type = 0; + prevdir.id = 0; + lfs->mlist = &prevdir; } - // create move to fix later - bool samepair = (lfs_pair_cmp(oldcwd.pair, newcwd.pair) == 0); - uint16_t newoldtagid = lfs_tag_id(oldtag); - if (samepair) { - // there is a small chance we are being renamed in the same - // directory/ to an id less than our old id, the global update - // to handle this is a bit messy - if (prevtag == LFS_ERR_NOENT && newid <= newoldtagid) { - newoldtagid += 1; - } - } else { - lfs_fs_prepmove(lfs, newoldtagid, oldcwd.pair); + if (!samepair) { + lfs_fs_prepmove(lfs, newoldid, oldcwd.pair); } // move over all attributes @@ -3250,8 +3282,9 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { {LFS_MKTAG(lfs_tag_type3(oldtag), newid, strlen(newpath)), newpath}, {LFS_MKTAG(LFS_FROM_MOVE, newid, lfs_tag_id(oldtag)), &oldcwd}, {LFS_MKTAG_IF(samepair, - LFS_TYPE_DELETE, newoldtagid, 0)})); + LFS_TYPE_DELETE, newoldid, 0)})); if (err) { + lfs->mlist = prevdir.next; LFS_TRACE("lfs_rename -> %d", err); return err; } @@ -3264,22 +3297,24 @@ int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) { err = lfs_dir_commit(lfs, &oldcwd, LFS_MKATTRS( {LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(oldtag), 0)})); if (err) { + lfs->mlist = prevdir.next; LFS_TRACE("lfs_rename -> %d", err); return err; } } + lfs->mlist = prevdir.next; if (prevtag != LFS_ERR_NOENT && lfs_tag_type3(prevtag) == LFS_TYPE_DIR) { // fix orphan lfs_fs_preporphans(lfs, -1); - err = lfs_fs_pred(lfs, prevdir.pair, &newcwd); + err = lfs_fs_pred(lfs, prevdir.m.pair, &newcwd); if (err) { LFS_TRACE("lfs_rename -> %d", err); return err; } - err = lfs_dir_drop(lfs, &newcwd, &prevdir); + err = lfs_dir_drop(lfs, &newcwd, &prevdir.m); if (err) { LFS_TRACE("lfs_rename -> %d", err); return err; @@ -3469,9 +3504,9 @@ static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) { lfs->root[1] = LFS_BLOCK_NULL; lfs->mlist = NULL; lfs->seed = 0; - lfs->gstate = (struct lfs_gstate){0}; - lfs->gpending = (struct lfs_gstate){0}; - lfs->gdelta = (struct lfs_gstate){0}; + lfs->gdisk = (lfs_gstate_t){0}; + lfs->gstate = (lfs_gstate_t){0}; + lfs->gdelta = (lfs_gstate_t){0}; #ifdef LFS_MIGRATE lfs->lfs1 = NULL; #endif @@ -3683,7 +3718,7 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { } // has gstate? - err = lfs_dir_getgstate(lfs, &dir, &lfs->gpending); + err = lfs_dir_getgstate(lfs, &dir, &lfs->gstate); if (err) { goto cleanup; } @@ -3696,14 +3731,14 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { } // update littlefs with gstate - lfs->gpending.tag += !lfs_tag_isvalid(lfs->gpending.tag); - lfs->gstate = lfs->gpending; - if (lfs_gstate_hasmove(&lfs->gstate)) { - LFS_DEBUG("Found move %"PRIx32" %"PRIx32" %"PRIx16, + if (!lfs_gstate_iszero(&lfs->gstate)) { + LFS_DEBUG("Found pending gstate %08"PRIx32" %08"PRIx32" %08"PRIx32, + lfs->gstate.tag, lfs->gstate.pair[0], - lfs->gstate.pair[1], - lfs_tag_id(lfs->gstate.tag)); + lfs->gstate.pair[1]); } + lfs->gstate.tag += !lfs_tag_isvalid(lfs->gstate.tag); + lfs->gdisk = lfs->gstate; // setup free lookahead lfs->free.off = lfs->seed % lfs->cfg->block_size; @@ -3920,8 +3955,8 @@ static int lfs_fs_relocate(lfs_t *lfs, // fix pending move in this pair? this looks like an optimization but // is in fact _required_ since relocating may outdate the move. uint16_t moveid = 0x3ff; - if (lfs_gstate_hasmovehere(&lfs->gpending, parent.pair)) { - moveid = lfs_tag_id(lfs->gpending.tag); + if (lfs_gstate_hasmovehere(&lfs->gstate, parent.pair)) { + moveid = lfs_tag_id(lfs->gstate.tag); LFS_DEBUG("Fixing move while relocating " "%"PRIx32" %"PRIx32" %"PRIx16"\n", parent.pair[0], parent.pair[1], moveid); @@ -3956,8 +3991,8 @@ static int lfs_fs_relocate(lfs_t *lfs, // fix pending move in this pair? this looks like an optimization but // is in fact _required_ since relocating may outdate the move. uint16_t moveid = 0x3ff; - if (lfs_gstate_hasmovehere(&lfs->gpending, parent.pair)) { - moveid = lfs_tag_id(lfs->gpending.tag); + if (lfs_gstate_hasmovehere(&lfs->gstate, parent.pair)) { + moveid = lfs_tag_id(lfs->gstate.tag); LFS_DEBUG("Fixing move while relocating " "%"PRIx32" %"PRIx32" %"PRIx16"\n", parent.pair[0], parent.pair[1], moveid); @@ -3980,40 +4015,40 @@ static int lfs_fs_relocate(lfs_t *lfs, } static void lfs_fs_preporphans(lfs_t *lfs, int8_t orphans) { - lfs->gpending.tag += orphans; - lfs_gstate_xororphans(&lfs->gdelta, &lfs->gpending, - lfs_gstate_hasorphans(&lfs->gpending)); - lfs_gstate_xororphans(&lfs->gpending, &lfs->gpending, - lfs_gstate_hasorphans(&lfs->gpending)); + LFS_ASSERT(lfs_tag_size(lfs->gstate.tag) > 0 || orphans >= 0); + lfs->gstate.tag += orphans; + lfs->gstate.tag = ((lfs->gstate.tag & ~LFS_MKTAG(0x800, 0, 0)) | + ((uint32_t)lfs_gstate_hasorphans(&lfs->gstate) << 31)); } static void lfs_fs_prepmove(lfs_t *lfs, uint16_t id, const lfs_block_t pair[2]) { - lfs_gstate_xormove(&lfs->gdelta, &lfs->gpending, id, pair); - lfs_gstate_xormove(&lfs->gpending, &lfs->gpending, id, pair); + lfs->gstate.tag = ((lfs->gstate.tag & ~LFS_MKTAG(0x7ff, 0x3ff, 0)) | + ((id != 0x3ff) ? LFS_MKTAG(LFS_TYPE_DELETE, id, 0) : 0)); + lfs->gstate.pair[0] = (id != 0x3ff) ? pair[0] : 0; + lfs->gstate.pair[1] = (id != 0x3ff) ? pair[1] : 0; } - static int lfs_fs_demove(lfs_t *lfs) { - if (!lfs_gstate_hasmove(&lfs->gstate)) { + if (!lfs_gstate_hasmove(&lfs->gdisk)) { return 0; } // Fix bad moves LFS_DEBUG("Fixing move %"PRIx32" %"PRIx32" %"PRIx16, - lfs->gstate.pair[0], - lfs->gstate.pair[1], - lfs_tag_id(lfs->gstate.tag)); + lfs->gdisk.pair[0], + lfs->gdisk.pair[1], + lfs_tag_id(lfs->gdisk.tag)); // fetch and delete the moved entry lfs_mdir_t movedir; - int err = lfs_dir_fetch(lfs, &movedir, lfs->gstate.pair); + int err = lfs_dir_fetch(lfs, &movedir, lfs->gdisk.pair); if (err) { return err; } // prep gstate and delete move id - uint16_t moveid = lfs_tag_id(lfs->gstate.tag); + uint16_t moveid = lfs_tag_id(lfs->gdisk.tag); lfs_fs_prepmove(lfs, 0x3ff, NULL); err = lfs_dir_commit(lfs, &movedir, LFS_MKATTRS( {LFS_MKTAG(LFS_TYPE_DELETE, moveid, 0)})); @@ -4030,12 +4065,12 @@ static int lfs_fs_deorphan(lfs_t *lfs) { } // Fix any orphans - lfs_mdir_t pdir = {.split = true}; - lfs_mdir_t dir = {.tail = {0, 1}}; + lfs_mdir_t pdir = {.split = true, .tail = {0, 1}}; + lfs_mdir_t dir; // iterate over all directory directory entries - while (!lfs_pair_isnull(dir.tail)) { - int err = lfs_dir_fetch(lfs, &dir, dir.tail); + while (!lfs_pair_isnull(pdir.tail)) { + int err = lfs_dir_fetch(lfs, &dir, pdir.tail); if (err) { return err; } @@ -4059,7 +4094,8 @@ static int lfs_fs_deorphan(lfs_t *lfs) { return err; } - break; + // refetch tail + continue; } lfs_block_t pair[2]; @@ -4072,8 +4108,9 @@ static int lfs_fs_deorphan(lfs_t *lfs) { if (!lfs_pair_sync(pair, pdir.tail)) { // we have desynced - LFS_DEBUG("Fixing half-orphan %"PRIx32" %"PRIx32, - pair[0], pair[1]); + LFS_DEBUG("Fixing half-orphan " + "%"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + pdir.tail[0], pdir.tail[1], pair[0], pair[1]); lfs_pair_tole32(pair); err = lfs_dir_commit(lfs, &pdir, LFS_MKATTRS( @@ -4083,16 +4120,16 @@ static int lfs_fs_deorphan(lfs_t *lfs) { return err; } - break; + // refetch tail + continue; } } - memcpy(&pdir, &dir, sizeof(pdir)); + pdir = dir; } // mark orphans as fixed lfs_fs_preporphans(lfs, -lfs_gstate_getorphans(&lfs->gstate)); - lfs->gstate = lfs->gpending; return 0; } diff --git a/lfs.h b/lfs.h index 04054e91..a7bd1863 100644 --- a/lfs.h +++ b/lfs.h @@ -355,6 +355,11 @@ typedef struct lfs_superblock { lfs_size_t attr_max; } lfs_superblock_t; +typedef struct lfs_gstate { + uint32_t tag; + lfs_block_t pair[2]; +} lfs_gstate_t; + // The littlefs filesystem type typedef struct lfs { lfs_cache_t rcache; @@ -369,10 +374,9 @@ typedef struct lfs { } *mlist; uint32_t seed; - struct lfs_gstate { - uint32_t tag; - lfs_block_t pair[2]; - } gstate, gpending, gdelta; + lfs_gstate_t gstate; + lfs_gstate_t gdisk; + lfs_gstate_t gdelta; struct lfs_free { lfs_block_t off; diff --git a/scripts/readtree.py b/scripts/readtree.py index 7d112fd9..30e3cfc0 100755 --- a/scripts/readtree.py +++ b/scripts/readtree.py @@ -197,16 +197,17 @@ def main(args): args.mdirs = True if args.superblock and superblock: - print("superblock %s" % json.dumps(superblock[0].data.decode('utf8'))) + print("superblock %s v%d.%d" % ( + json.dumps(superblock[0].data.decode('utf8')), + struct.unpack(' 0; ''' +[[case]] # noop move, yes this is legal +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "hi") => 0; + lfs_rename(&lfs, "hi", "hi") => 0; + lfs_mkdir(&lfs, "hi/hi") => 0; + lfs_rename(&lfs, "hi/hi", "hi/hi") => 0; + lfs_mkdir(&lfs, "hi/hi/hi") => 0; + lfs_rename(&lfs, "hi/hi/hi", "hi/hi/hi") => 0; + lfs_stat(&lfs, "hi/hi/hi", &info) => 0; + assert(strcmp(info.name, "hi") == 0); + assert(info.type == LFS_TYPE_DIR); + lfs_unmount(&lfs) => 0; +''' + [[case]] # move file corrupt source in = "lfs.c" code = ''' diff --git a/tests_/test_orphan.toml b/tests_/test_orphans.toml similarity index 50% rename from tests_/test_orphan.toml rename to tests_/test_orphans.toml index fe521992..3bf0454e 100644 --- a/tests_/test_orphan.toml +++ b/tests_/test_orphans.toml @@ -54,3 +54,64 @@ code = ''' lfs_fs_size(&lfs) => 8; lfs_unmount(&lfs) => 0; ''' + +[[case]] # reentrant testing for orphans, basically just spam mkdir/remove +reentrant = true +define = [ + {FILES=6, DEPTH=1, CYCLES=50}, + {FILES=26, DEPTH=1, CYCLES=50}, + {FILES=3, DEPTH=3, CYCLES=50}, +] +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + srand(1); + const char alpha[] = "abcdefghijklmnopqrstuvwxyz"; + for (int i = 0; i < CYCLES; i++) { + // create random path + char full_path[256]; + for (int d = 0; d < DEPTH; d++) { + sprintf(&full_path[2*d], "/%c", alpha[rand() % FILES]); + } + + // if it does not exist, we create it, else we destroy + int res = lfs_stat(&lfs, full_path, &info); + if (res == LFS_ERR_NOENT) { + // create each directory in turn, ignore if dir already exists + for (int d = 0; d < DEPTH; d++) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + err = lfs_mkdir(&lfs, path); + assert(!err || err == LFS_ERR_EXIST); + } + + for (int d = 0; d < DEPTH; d++) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + lfs_stat(&lfs, path, &info) => 0; + assert(strcmp(info.name, &path[2*d+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + } + } else { + // is valid dir? + assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + + // try to delete path in reverse order, ignore if dir is not empty + for (int d = DEPTH-1; d >= 0; d--) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + err = lfs_remove(&lfs, path); + assert(!err || err == LFS_ERR_NOTEMPTY); + } + + lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT; + } + } + lfs_unmount(&lfs) => 0; +''' + diff --git a/tests_/test_relocations.toml b/tests_/test_relocations.toml index 1fa73e51..ad267b51 100644 --- a/tests_/test_relocations.toml +++ b/tests_/test_relocations.toml @@ -1,8 +1,8 @@ # specific corner cases worth explicitly testing for - [[case]] # dangling split dir test define.ITERATIONS = 20 define.COUNT = 10 +define.LFS_BLOCK_CYCLES = [8, 1] code = ''' lfs_format(&lfs, &cfg) => 0; // fill up filesystem so only ~16 blocks are left @@ -68,6 +68,7 @@ code = ''' [[case]] # outdated head test define.ITERATIONS = 20 define.COUNT = 10 +define.LFS_BLOCK_CYCLES = [8, 1] code = ''' lfs_format(&lfs, &cfg) => 0; // fill up filesystem so only ~16 blocks are left @@ -141,3 +142,160 @@ code = ''' } lfs_unmount(&lfs) => 0; ''' + +[[case]] # reentrant testing for relocations, this is the same as the + # orphan testing, except here we also set block_cycles so that + # almost every tree operation needs a relocation +reentrant = true +define = [ + {FILES=6, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, + {FILES=26, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, + {FILES=3, DEPTH=3, CYCLES=50, LFS_BLOCK_CYCLES=1}, +] +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + srand(1); + const char alpha[] = "abcdefghijklmnopqrstuvwxyz"; + for (int i = 0; i < CYCLES; i++) { + // create random path + char full_path[256]; + for (int d = 0; d < DEPTH; d++) { + sprintf(&full_path[2*d], "/%c", alpha[rand() % FILES]); + } + + // if it does not exist, we create it, else we destroy + int res = lfs_stat(&lfs, full_path, &info); + if (res == LFS_ERR_NOENT) { + // create each directory in turn, ignore if dir already exists + for (int d = 0; d < DEPTH; d++) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + err = lfs_mkdir(&lfs, path); + assert(!err || err == LFS_ERR_EXIST); + } + + for (int d = 0; d < DEPTH; d++) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + lfs_stat(&lfs, path, &info) => 0; + assert(strcmp(info.name, &path[2*d+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + } + } else { + // is valid dir? + assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + + // try to delete path in reverse order, ignore if dir is not empty + for (int d = DEPTH-1; d >= 0; d--) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + err = lfs_remove(&lfs, path); + assert(!err || err == LFS_ERR_NOTEMPTY); + } + + lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT; + } + } + lfs_unmount(&lfs) => 0; +''' + +[[case]] # reentrant testing for relocations, but now with random renames! +reentrant = true +define = [ + {FILES=6, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, + {FILES=26, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, + {FILES=3, DEPTH=3, CYCLES=50, LFS_BLOCK_CYCLES=1}, +] +code = ''' + err = lfs_mount(&lfs, &cfg); + if (err) { + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + } + + srand(1); + const char alpha[] = "abcdefghijklmnopqrstuvwxyz"; + for (int i = 0; i < CYCLES; i++) { + // create random path + char full_path[256]; + for (int d = 0; d < DEPTH; d++) { + sprintf(&full_path[2*d], "/%c", alpha[rand() % FILES]); + } + + // if it does not exist, we create it, else we destroy + int res = lfs_stat(&lfs, full_path, &info); + assert(!res || res == LFS_ERR_NOENT); + if (res == LFS_ERR_NOENT) { + // create each directory in turn, ignore if dir already exists + for (int d = 0; d < DEPTH; d++) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + err = lfs_mkdir(&lfs, path); + assert(!err || err == LFS_ERR_EXIST); + } + + for (int d = 0; d < DEPTH; d++) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + lfs_stat(&lfs, path, &info) => 0; + assert(strcmp(info.name, &path[2*d+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + } + } else { + assert(strcmp(info.name, &full_path[2*(DEPTH-1)+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + + // create new random path + char new_path[256]; + for (int d = 0; d < DEPTH; d++) { + sprintf(&new_path[2*d], "/%c", alpha[rand() % FILES]); + } + + // if new path does not exist, rename, otherwise destroy + res = lfs_stat(&lfs, new_path, &info); + assert(!res || res == LFS_ERR_NOENT); + if (res == LFS_ERR_NOENT) { + // stop once some dir is renamed + for (int d = 0; d < DEPTH; d++) { + strcpy(&path[2*d], &full_path[2*d]); + path[2*d+2] = '\0'; + strcpy(&path[128+2*d], &new_path[2*d]); + path[128+2*d+2] = '\0'; + err = lfs_rename(&lfs, path, path+128); + assert(!err || err == LFS_ERR_NOTEMPTY); + if (!err) { + strcpy(path, path+128); + } + } + + for (int d = 0; d < DEPTH; d++) { + strcpy(path, new_path); + path[2*d+2] = '\0'; + lfs_stat(&lfs, path, &info) => 0; + assert(strcmp(info.name, &path[2*d+1]) == 0); + assert(info.type == LFS_TYPE_DIR); + } + + lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT; + } else { + // try to delete path in reverse order, + // ignore if dir is not empty + for (int d = DEPTH-1; d >= 0; d--) { + strcpy(path, full_path); + path[2*d+2] = '\0'; + err = lfs_remove(&lfs, path); + assert(!err || err == LFS_ERR_NOTEMPTY); + } + + lfs_stat(&lfs, full_path, &info) => LFS_ERR_NOENT; + } + } + } + lfs_unmount(&lfs) => 0; +''' From b9d0695e0a026d144772480f339e577782729949 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sat, 25 Jan 2020 15:59:17 -0600 Subject: [PATCH 13/41] Rewrote explode_asserts.py to be more efficient Normally I wouldn't consider optimizing this sort of script, but explode_asserts.py proved to be terribly inefficient and dominated the build time for running tests. It was slow enough to be distracting when attempting to test patches while debugging. Just running explode_asserts.py was ~10x slower than the rest of the compilation process. After implementing a proper tokenizer and switching to a handwritten recursive descent parser, I was able to speed up explode_asserts.py by ~5x and make test compilation much more tolerable. I don't think this was a limitaiton of parsy, but rather switching to a recursive descent parser made it much easier to find the hotspots where parsing was wasting cycles (string slicing for one). It's interesting to note that while the assert patterns can be parsed with a LL(1) parser (by dumping seen tokens if a pattern fails), I didn't bother as it's much easier to write the patterns with LL(k) and parsing asserts is predicated by the "assert" string. A few other tweaks: - allowed combining different test modes in one run - added a --no-internal option - changed test_.py to start counting cases from 1 - added assert(memcmp(a, b) == 0) matching - added better handling of string escapes in assert messages time to run tests: before: 1m31.122s after: 0m41.447s --- scripts/explode_asserts.py | 536 ++++++++++++++++++++++++------------- scripts/test_.py | 105 ++++---- 2 files changed, 404 insertions(+), 237 deletions(-) diff --git a/scripts/explode_asserts.py b/scripts/explode_asserts.py index ff3f260a..c0534cb1 100755 --- a/scripts/explode_asserts.py +++ b/scripts/explode_asserts.py @@ -1,208 +1,372 @@ #!/usr/bin/env python3 -import parsy as p import re -import io import sys -ASSERT_PATTERN = p.string('LFS_ASSERT') | p.string('assert') -ASSERT_CHARS = 'La' -ASSERT_TARGET = '__LFS_ASSERT_{TYPE}_{COMP}' -ASSERT_TESTS = { - 'int': """ - __typeof__({lh}) _lh = {lh}; - __typeof__({lh}) _rh = (__typeof__({lh})){rh}; - if (!(_lh {op} _rh)) {{ - printf("%s:%d:assert: " - "assert failed with %"PRIiMAX", expected {comp} %"PRIiMAX"\\n", - {file}, {line}, (intmax_t)_lh, (intmax_t)_rh); - fflush(NULL); - raise(SIGABRT); +PATTERN = ['LFS_ASSERT', 'assert'] +PREFIX = 'LFS' +MAXWIDTH = 16 + +ASSERT = "__{PREFIX}_ASSERT_{TYPE}_{COMP}" +FAIL = """ +__attribute__((unused)) +static void __{prefix}_assert_fail_{type}( + const char *file, int line, const char *comp, + {ctype} lh, size_t lsize, + {ctype} rh, size_t rsize) {{ + printf("%s:%d:assert: assert failed with ", file, line); + __{prefix}_assert_print_{type}(lh, lsize); + printf(", expected %s ", comp); + __{prefix}_assert_print_{type}(rh, rsize); + printf("\\n"); + fflush(NULL); + raise(SIGABRT); +}} +""" + +COMP = { + '==': 'eq', + '!=': 'ne', + '<=': 'le', + '>=': 'ge', + '<': 'lt', + '>': 'gt', +} + +TYPE = { + 'int': { + 'ctype': 'intmax_t', + 'fail': FAIL, + 'print': """ + __attribute__((unused)) + static void __{prefix}_assert_print_{type}({ctype} v, size_t size) {{ + (void)size; + printf("%"PRIiMAX, v); + }} + """, + 'assert': """ + #define __{PREFIX}_ASSERT_{TYPE}_{COMP}(file, line, lh, rh) + do {{ + __typeof__(lh) _lh = lh; + __typeof__(lh) _rh = (__typeof__(lh))rh; + if (!(_lh {op} _rh)) {{ + __{prefix}_assert_fail_{type}(file, line, "{comp}", + (intmax_t)_lh, 0, (intmax_t)_rh, 0); + }} + }} while (0) + """ + }, + 'bool': { + 'ctype': 'bool', + 'fail': FAIL, + 'print': """ + __attribute__((unused)) + static void __{prefix}_assert_print_{type}({ctype} v, size_t size) {{ + (void)size; + printf("%s", v ? "true" : "false"); }} - """, - 'str': """ - const char *_lh = {lh}; - const char *_rh = {rh}; - if (!(strcmp(_lh, _rh) {op} 0)) {{ - printf("%s:%d:assert: " - "assert failed with \\\"%s\\\", expected {comp} \\\"%s\\\"\\n", - {file}, {line}, _lh, _rh); - fflush(NULL); - raise(SIGABRT); + """, + 'assert': """ + #define __{PREFIX}_ASSERT_{TYPE}_{COMP}(file, line, lh, rh) + do {{ + bool _lh = !!(lh); + bool _rh = !!(rh); + if (!(_lh {op} _rh)) {{ + __{prefix}_assert_fail_{type}(file, line, "{comp}", + _lh, 0, _rh, 0); + }} + }} while (0) + """ + }, + 'mem': { + 'ctype': 'const void *', + 'fail': FAIL, + 'print': """ + __attribute__((unused)) + static void __{prefix}_assert_print_{type}({ctype} v, size_t size) {{ + const uint8_t *s = v; + printf("\\\""); + for (size_t i = 0; i < size && i < {maxwidth}; i++) {{ + if (s[i] >= ' ' && s[i] <= '~') {{ + printf("%c", s[i]); + }} else {{ + printf("\\\\x%02x", s[i]); + }} + }} + if (size > {maxwidth}) {{ + printf("..."); + }} + printf("\\\""); }} - """, - 'bool': """ - bool _lh = !!({lh}); - bool _rh = !!({rh}); - if (!(_lh {op} _rh)) {{ - printf("%s:%d:assert: " - "assert failed with %s, expected {comp} %s\\n", - {file}, {line}, _lh ? "true" : "false", _rh ? "true" : "false"); - fflush(NULL); - raise(SIGABRT); + """, + 'assert': """ + #define __{PREFIX}_ASSERT_{TYPE}_{COMP}(file, line, lh, rh, size) + do {{ + const void *_lh = lh; + const void *_rh = rh; + if (!(memcmp(_lh, _rh, size) {op} 0)) {{ + __{prefix}_assert_fail_{type}(file, line, "{comp}", + _lh, size, _rh, size); + }} + }} while (0) + """ + }, + 'str': { + 'ctype': 'const char *', + 'fail': FAIL, + 'print': """ + __attribute__((unused)) + static void __{prefix}_assert_print_{type}({ctype} v, size_t size) {{ + __{prefix}_assert_print_mem(v, size); }} - """, + """, + 'assert': """ + #define __{PREFIX}_ASSERT_{TYPE}_{COMP}(file, line, lh, rh) + do {{ + const char *_lh = lh; + const char *_rh = rh; + if (!(strcmp(_lh, _rh) {op} 0)) {{ + __{prefix}_assert_fail_{type}(file, line, "{comp}", + _lh, strlen(_lh), _rh, strlen(_rh)); + }} + }} while (0) + """ + } } -def mkassert(lh, rh='true', type='bool', comp='eq'): - return ((ASSERT_TARGET + "({lh}, {rh}, __FILE__, __LINE__, __func__)") - .format( - type=type, TYPE=type.upper(), - comp=comp, COMP=comp.upper(), - lh=lh.strip(' '), - rh=rh.strip(' '))) - -def mkdecl(type, comp, op): - return (( - "#define "+ASSERT_TARGET+"(lh, rh, file, line, func)" - " do {{"+re.sub('\s+', ' ', ASSERT_TESTS[type])+"}} while (0)\n") - .format( - type=type, TYPE=type.upper(), - comp=comp, COMP=comp.upper(), - lh='lh', rh='rh', op=op, - file='file', line='line', func='func')) - -# add custom until combinator -def until(self, end): - return end.should_fail('should fail').then(self).many() -p.Parser.until = until - -pcomp = ( - p.string('==').tag('eq') | - p.string('!=').tag('ne') | - p.string('<=').tag('le') | - p.string('>=').tag('ge') | - p.string('<').tag('lt') | - p.string('>').tag('gt')); - -plogic = p.string('&&') | p.string('||') - -@p.generate -def pstrassert(): - yield ASSERT_PATTERN + p.regex('\s*') + p.string('(') + p.regex('\s*') - yield p.string('strcmp') + p.regex('\s*') + p.string('(') + p.regex('\s*') - lh = yield pexpr.until(p.string(',') | p.string(')') | plogic) - yield p.string(',') + p.regex('\s*') - rh = yield pexpr.until(p.string(')') | plogic) - yield p.string(')') + p.regex('\s*') - op = yield pcomp - yield p.regex('\s*') + p.string('0') + p.regex('\s*') + p.string(')') - return mkassert(''.join(lh), ''.join(rh), 'str', op[0]) - -@p.generate -def pintassert(): - yield ASSERT_PATTERN + p.regex('\s*') + p.string('(') + p.regex('\s*') - lh = yield pexpr.until(pcomp | p.string(')') | plogic) - op = yield pcomp - rh = yield pexpr.until(p.string(')') | plogic) - yield p.string(')') - return mkassert(''.join(lh), ''.join(rh), 'int', op[0]) - -@p.generate -def pboolassert(): - yield ASSERT_PATTERN + p.regex('\s*') + p.string('(') + p.regex('\s*') - expr = yield pexpr.until(p.string(')')) - yield p.string(')') - return mkassert(''.join(expr), 'true', 'bool', 'eq') - -passert = p.peek(ASSERT_PATTERN) >> (pstrassert | pintassert | pboolassert) - -@p.generate -def pcomment1(): - yield p.string('//') - s = yield p.regex('[^\\n]*') - yield p.string('\n') - return '//' + s + '\n' - -@p.generate -def pcomment2(): - yield p.string('/*') - s = yield p.regex('((?!\*/).)*') - yield p.string('*/') - return '/*' + ''.join(s) + '*/' - -@p.generate -def pcomment3(): - yield p.string('#') - s = yield p.regex('[^\\n]*') - yield p.string('\n') - return '#' + s + '\n' - -pws = p.regex('\s+') | pcomment1 | pcomment2 | pcomment3 - -@p.generate -def pstring(): - q = yield p.regex('["\']') - s = yield (p.string('\\%s' % q) | p.regex('[^%s]' % q)).many() - yield p.string(q) - return q + ''.join(s) + q - -@p.generate -def pnested(): - l = yield p.string('(') - n = yield pexpr.until(p.string(')')) - r = yield p.string(')') - return l + ''.join(n) + r - -pexpr = ( - # shortcut for a bit better performance - p.regex('[^%s/#\'"():;{}=><,&|-]+' % ASSERT_CHARS) | - pws | - passert | - pstring | - pnested | - p.string('->') | - p.regex('.', re.DOTALL)) - -@p.generate -def pstmt(): - ws = yield pws.many() - lh = yield pexpr.until(p.string('=>') | p.regex('[:;{}]')) - op = yield p.string('=>').optional() - if op == '=>': - rh = yield pstmt - return ''.join(ws) + mkassert(''.join(lh), rh, 'int', 'eq') +def mkdecls(outf, maxwidth=16): + outf.write("#include \n") + outf.write("#include \n") + outf.write("#include \n") + outf.write("#include \n") + outf.write("#include \n") + + for type, desc in sorted(TYPE.items()): + format = { + 'type': type.lower(), 'TYPE': type.upper(), + 'ctype': desc['ctype'], + 'prefix': PREFIX.lower(), 'PREFIX': PREFIX.upper(), + 'maxwidth': maxwidth, + } + outf.write(re.sub('\s+', ' ', + desc['print'].strip().format(**format))+'\n') + outf.write(re.sub('\s+', ' ', + desc['fail'].strip().format(**format))+'\n') + + for op, comp in sorted(COMP.items()): + format.update({ + 'comp': comp.lower(), 'COMP': comp.upper(), + 'op': op, + }) + outf.write(re.sub('\s+', ' ', + desc['assert'].strip().format(**format))+'\n') + +def mkassert(type, comp, lh, rh, size=None): + format = { + 'type': type.lower(), 'TYPE': type.upper(), + 'comp': comp.lower(), 'COMP': comp.upper(), + 'prefix': PREFIX.lower(), 'PREFIX': PREFIX.upper(), + 'lh': lh.strip(), + 'rh': rh.strip(), + 'size': size, + } + if size: + return ((ASSERT + '(__FILE__, __LINE__, {lh}, {rh}, {size})') + .format(**format)) else: - return ''.join(ws) + ''.join(lh) + return ((ASSERT + '(__FILE__, __LINE__, {lh}, {rh})') + .format(**format)) + + +# simple recursive descent parser +LEX = { + 'ws': [r'(?:\s|\n|#.*?\n|//.*?\n|/\*.*?\*/)+'], + 'assert': PATTERN, + 'string': [r'"(?:\\.|[^"])*"', r"'(?:\\.|[^'])\'"], + 'arrow': ['=>'], + 'paren': ['\(', '\)'], + 'op': ['strcmp', 'memcmp', '->'], + 'comp': ['==', '!=', '<=', '>=', '<', '>'], + 'logic': ['\&\&', '\|\|'], + 'sep': [':', ';', '\{', '\}', ','], +} + +class ParseFailure(Exception): + def __init__(self, expected, found): + self.expected = expected + self.found = found + + def __str__(self): + return "expected %r, found %s..." % ( + self.expected, repr(self.found)[:70]) + +class Parse: + def __init__(self, inf, lexemes): + p = '|'.join('(?P<%s>%s)' % (n, '|'.join(l)) + for n, l in lexemes.items()) + p = re.compile(p, re.DOTALL) + data = inf.read() + tokens = [] + while True: + m = p.search(data) + if m: + if m.start() > 0: + tokens.append((None, data[:m.start()])) + tokens.append((m.lastgroup, m.group())) + data = data[m.end():] + else: + tokens.append((None, data)) + break + self.tokens = tokens + self.off = 0 + + def lookahead(self, *pattern): + if self.off < len(self.tokens): + token = self.tokens[self.off] + if token[0] in pattern or token[1] in pattern: + self.m = token[1] + return self.m + self.m = None + return self.m + + def accept(self, *patterns): + m = self.lookahead(*patterns) + if m is not None: + self.off += 1 + return m + + def expect(self, *patterns): + m = self.accept(*patterns) + if not m: + raise ParseFailure(patterns, self.tokens[self.off:]) + return m + + def push(self): + return self.off + + def pop(self, state): + self.off = state + +def passert(p): + def pastr(p): + p.expect('assert') ; p.accept('ws') ; p.expect('(') ; p.accept('ws') + p.expect('strcmp') ; p.accept('ws') ; p.expect('(') ; p.accept('ws') + lh = pexpr(p) ; p.accept('ws') + p.expect(',') ; p.accept('ws') + rh = pexpr(p) ; p.accept('ws') + p.expect(')') ; p.accept('ws') + comp = p.expect('comp') ; p.accept('ws') + p.expect('0') ; p.accept('ws') + p.expect(')') + return mkassert('str', COMP[comp], lh, rh) + + def pamem(p): + p.expect('assert') ; p.accept('ws') ; p.expect('(') ; p.accept('ws') + p.expect('memcmp') ; p.accept('ws') ; p.expect('(') ; p.accept('ws') + lh = pexpr(p) ; p.accept('ws') + p.expect(',') ; p.accept('ws') + rh = pexpr(p) ; p.accept('ws') + p.expect(',') ; p.accept('ws') + size = pexpr(p) ; p.accept('ws') + p.expect(')') ; p.accept('ws') + comp = p.expect('comp') ; p.accept('ws') + p.expect('0') ; p.accept('ws') + p.expect(')') + return mkassert('mem', COMP[comp], lh, rh, size) + + def paint(p): + p.expect('assert') ; p.accept('ws') ; p.expect('(') ; p.accept('ws') + lh = pexpr(p) ; p.accept('ws') + comp = p.expect('comp') ; p.accept('ws') + rh = pexpr(p) ; p.accept('ws') + p.expect(')') + return mkassert('int', COMP[comp], lh, rh) + + def pabool(p): + p.expect('assert') ; p.accept('ws') ; p.expect('(') ; p.accept('ws') + lh = pexprs(p) ; p.accept('ws') + p.expect(')') + return mkassert('bool', 'eq', lh, 'true') + + def pa(p): + return p.expect('assert') + + state = p.push() + lastf = None + for pa in [pastr, pamem, paint, pabool, pa]: + try: + return pa(p) + except ParseFailure as f: + p.pop(state) + lastf = f + else: + raise lastf + +def pexpr(p): + res = [] + while True: + if p.accept('('): + res.append(p.m) + while True: + res.append(pexprs(p)) + if p.accept('sep'): + res.append(p.m) + else: + break + res.append(p.expect(')')) + elif p.lookahead('assert'): + res.append(passert(p)) + elif p.accept('assert', 'ws', 'string', 'op', None): + res.append(p.m) + else: + return ''.join(res) + +def pexprs(p): + res = [] + while True: + res.append(pexpr(p)) + if p.accept('comp', 'logic', ','): + res.append(p.m) + else: + return ''.join(res) + +def pstmt(p): + ws = p.accept('ws') or '' + lh = pexprs(p) + if p.accept('=>'): + rh = pexprs(p) + return ws + mkassert('int', 'eq', lh, rh) + else: + return ws + lh -@p.generate -def pstmts(): - a = yield pstmt - b = yield (p.regex('[:;{}]') + pstmt).many() - return [a] + b def main(args): inf = open(args.input, 'r') if args.input else sys.stdin outf = open(args.output, 'w') if args.output else sys.stdout - # parse C code - input = inf.read() - stmts = pstmts.parse(input) + lexemes = LEX.copy() + if args.pattern: + lexemes['assert'] = args.pattern + p = Parse(inf, lexemes) # write extra verbose asserts - outf.write("#include \n") - outf.write("#include \n") - outf.write("#include \n") - outf.write("#include \n") - outf.write(mkdecl('int', 'eq', '==')) - outf.write(mkdecl('int', 'ne', '!=')) - outf.write(mkdecl('int', 'lt', '<')) - outf.write(mkdecl('int', 'gt', '>')) - outf.write(mkdecl('int', 'le', '<=')) - outf.write(mkdecl('int', 'ge', '>=')) - outf.write(mkdecl('str', 'eq', '==')) - outf.write(mkdecl('str', 'ne', '!=')) - outf.write(mkdecl('str', 'lt', '<')) - outf.write(mkdecl('str', 'gt', '>')) - outf.write(mkdecl('str', 'le', '<=')) - outf.write(mkdecl('str', 'ge', '>=')) - outf.write(mkdecl('bool', 'eq', '==')) + mkdecls(outf, maxwidth=args.maxwidth) if args.input: outf.write("#line %d \"%s\"\n" % (1, args.input)) - # write parsed statements - for stmt in stmts: - outf.write(stmt) + # parse and write out stmt at a time + try: + while True: + outf.write(pstmt(p)) + if p.accept('sep'): + outf.write(p.m) + else: + break + except ParseFailure as f: + pass + + for i in range(p.off, len(p.tokens)): + outf.write(p.tokens[i][1]) if __name__ == "__main__": import argparse @@ -210,6 +374,10 @@ def main(args): description="Cpp step that increases assert verbosity") parser.add_argument('input', nargs='?', help="Input C file after cpp.") - parser.add_argument('-o', '--output', + parser.add_argument('-o', '--output', required=True, help="Output C file.") + parser.add_argument('-p', '--pattern', action='append', + help="Patterns to search for starting an assert statement.") + parser.add_argument('--maxwidth', default=MAXWIDTH, type=int, + help="Maximum number of characters to display for strcmp and memcmp.") main(parser.parse_args()) diff --git a/scripts/test_.py b/scripts/test_.py index b66ad561..1a13c5b1 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -4,24 +4,6 @@ # .toml files stored in the tests directory. # -# TODO -# x nargs > 1? -# x show perm config on failure -# x filtering -# n show perm config on verbose? -# x better lineno tracking for cases? -# n non-int perms? -# x different path format? -# - suite.prologue, suite.epilogue -# x in -# x change BLOCK_CYCLES to -1 by default -# x change persist behaviour -# x config chaining correct -# - why can't gdb see my defines? -# - say no to internal? -# x buffering stdout issues? -# - debug fast - import toml import glob import re @@ -125,6 +107,7 @@ def __init__(self, case, returncode=None, stdout=None, assert_=None): class TestCase: def __init__(self, config, filter=filter, suite=None, caseno=None, lineno=None, **_): + self.config = config self.filter = filter self.suite = suite self.caseno = caseno @@ -150,8 +133,10 @@ def __str__(self): return '%s#%d' % ( self.suite.name, self.caseno) - def permute(self, defines, permno=None, **_): - ncase = copy.copy(self) + def permute(self, class_=None, defines={}, permno=None, **_): + ncase = (class_ or type(self))(self.config) + for k, v in self.__dict__.items(): + setattr(ncase, k, v) ncase.case = self ncase.perms = [ncase] ncase.permno = permno @@ -194,6 +179,8 @@ def shouldtest(self, **args): len(self.filter) >= 2 and self.filter[1] != self.permno): return False + elif args.get('no_internal', False) and self.in_ is not None: + return False elif self.if_ is not None: return eval(self.if_, None, self.defines.copy()) else: @@ -210,6 +197,8 @@ def test(self, exec=[], persist=False, cycles=None, if persist != 'noerase': try: os.remove(self.suite.path + '.disk') + if args.get('verbose', False): + print('rm', self.suite.path + '.disk') except FileNotFoundError: pass @@ -225,7 +214,7 @@ def test(self, exec=[], persist=False, cycles=None, if gdb == 'assert': ncmd.extend(['-ex', 'r']) if failure.assert_: - ncmd.extend(['-ex', 'up']) + ncmd.extend(['-ex', 'up 2']) elif gdb == 'start': ncmd.extend([ '-ex', 'b %s:%d' % (self.suite.path, self.code_lineno), @@ -331,13 +320,15 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): raise class TestSuite: - def __init__(self, path, filter=None, TestCase=TestCase, **args): + def __init__(self, path, classes=[TestCase], defines={}, + filter=None, **args): self.name = os.path.basename(path) if self.name.endswith('.toml'): self.name = self.name[:-len('.toml')] self.path = path + self.classes = classes + self.defines = defines self.filter = filter - self.TestCase = TestCase with open(path) as f: # load tests @@ -356,7 +347,9 @@ def __init__(self, path, filter=None, TestCase=TestCase, **args): code_linenos.reverse() # grab global config - self.defines = config.get('define', {}) + for k, v in config.get('define', {}).items(): + if k not in self.defines: + self.defines[k] = v self.code = config.get('code', None) if self.code is not None: self.code_lineno = code_linenos.pop() @@ -372,8 +365,8 @@ def __init__(self, path, filter=None, TestCase=TestCase, **args): if k not in case: case[k] = v # initialize test case - self.cases.append(self.TestCase(case, filter=filter, - suite=self, caseno=i, lineno=lineno, **args)) + self.cases.append(TestCase(case, filter=filter, + suite=self, caseno=i+1, lineno=lineno, **args)) def __str__(self): return self.name @@ -381,14 +374,14 @@ def __str__(self): def __lt__(self, other): return self.name < other.name - def permute(self, defines={}, **args): + def permute(self, **args): for case in self.cases: # lets find all parameterized definitions, in one of [args.D, # suite.defines, case.defines, DEFINES]. Note that each of these # can be either a dict of defines, or a list of dicts, expressing # an initial set of permutations. pending = [{}] - for inits in [defines, self.defines, case.defines, DEFINES]: + for inits in [self.defines, case.defines, DEFINES]: if not isinstance(inits, list): inits = [inits] @@ -422,8 +415,10 @@ def permute(self, defines={}, **args): # generate permutations case.perms = [] - for i, perm in enumerate(expanded): - case.perms.append(case.permute(perm, permno=i, **args)) + for i, (class_, defines) in enumerate( + it.product(self.classes, expanded)): + case.perms.append(case.permute( + class_, defines, permno=i+1, **args)) # also track non-unique defines case.defines = {} @@ -519,16 +514,12 @@ def build(self, **args): self.target = self.path + '.test' return self.makefile, self.target - def test(self, caseno=None, permno=None, **args): + def test(self, **args): # run test suite! if not args.get('verbose', True): sys.stdout.write(self.name + ' ') sys.stdout.flush() for perm in self.perms: - if caseno is not None and perm.caseno != caseno: - continue - if permno is not None and perm.permno != permno: - continue if not perm.shouldtest(**args): continue @@ -553,6 +544,23 @@ def test(self, caseno=None, permno=None, **args): sys.stdout.write('\n') def main(**args): + # figure out explicit defines + defines = {} + for define in args['D']: + k, v, *_ = define.split('=', 2) + [''] + defines[k] = v + + # and what class of TestCase to run + classes = [] + if args.get('normal', False): + classes.append(TestCase) + if args.get('reentrant', False): + classes.append(ReentrantTestCase) + if args.get('valgrind', False): + classes.append(ValgrindTestCase) + if not classes: + classes = [TestCase] + suites = [] for testpath in args['testpaths']: # optionally specified test case/perm @@ -571,27 +579,14 @@ def main(**args): # find tests for path in glob.glob(testpath): - if args.get('valgrind', False): - TestCase_ = ValgrindTestCase - elif args.get('reentrant', False): - TestCase_ = ReentrantTestCase - else: - TestCase_ = TestCase - - suites.append(TestSuite(path, - filter=filter, TestCase=TestCase_, **args)) + suites.append(TestSuite(path, classes, defines, filter, **args)) # sort for reproducability suites = sorted(suites) # generate permutations - defines = {} - for define in args['D']: - k, v, *_ = define.split('=', 2) + [''] - defines[k] = v - for suite in suites: - suite.permute(defines, **args) + suite.permute(**args) # build tests in parallel print('====== building ======') @@ -736,10 +731,14 @@ def main(**args): parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], nargs='?', const='assert', help="Drop into gdb on test failure.") - parser.add_argument('--valgrind', action='store_true', - help="Run non-leaky tests under valgrind to check for memory leaks.") - parser.add_argument('--reentrant', action='store_true', + parser.add_argument('--no-internal', action='store_true', + help="Don't run tests that require internal knowledge.") + parser.add_argument('-n', '--normal', action='store_true', + help="Run tests normally.") + parser.add_argument('-r', '--reentrant', action='store_true', help="Run reentrant tests with simulated power-loss.") + parser.add_argument('-V', '--valgrind', action='store_true', + help="Run non-leaky tests under valgrind to check for memory leaks.") parser.add_argument('-e', '--exec', default=[], type=lambda e: e.split(' '), help="Run tests with another executable prefixed on the command line.") sys.exit(main(**vars(parser.parse_args()))) From 52ef0c1c9efac025b71f2bf3d33e8785538e88c7 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 26 Jan 2020 19:16:57 -0600 Subject: [PATCH 14/41] Fixed a crazy consistency issue in test.py The root of the problem was the notorious Python quirk with mutable default parameters. The default defines for the TestSuite class ended up being mutated as the class determined the permutations to test, corrupting other test's defines. However, the only define that was mutated this way was the CACHE_SIZE config in test_entries. The crazy thing was how this small innocuous change would cause "./scripts/test.py -nr test_relocations" and "./scripts/test.py -nr" to drift out of sync only after a commit spanning the different cache sizes would be written out with a different number of prog calls. This offset the power-cycle counter enough to cause one case to make it to an erase, and the other to not. Normally, the difference between a successful/unsuccessful erase wouldn't change the result of a test, but in this case it offset the revision count used for wear-leveling, causing one run run expand the superblock and the other to not. This change to the filesystem would then propogate through the rest of the test, making it difficult to reproduce test failures. Fortunately the fix was to just make a copy of the default define dictionary. This should also prevent accidently mutating of dicts belonging to our caller. Oh, also fixed a buffer overflow in test_files. --- scripts/test_.py | 6 +++--- testbd/lfs_testbd.c | 4 ++-- tests_/test_files.toml | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/scripts/test_.py b/scripts/test_.py index 1a13c5b1..4593b40a 100755 --- a/scripts/test_.py +++ b/scripts/test_.py @@ -304,8 +304,8 @@ def test(self, exec=[], persist=False, gdb=False, failure=None, **args): # exact cycle we should drop into debugger? if gdb and failure and failure.cycleno == cycles: - return super().test(gdb=gdb, - persist=persist, failure=failure, **args) + return super().test(gdb=gdb, persist=persist, cycles=cycles, + failure=failure, **args) # run tests, but kill the program after prog/erase has # been hit n cycles. We exit with a special return code if the @@ -327,7 +327,7 @@ def __init__(self, path, classes=[TestCase], defines={}, self.name = self.name[:-len('.toml')] self.path = path self.classes = classes - self.defines = defines + self.defines = defines.copy() self.filter = filter with open(path) as f: diff --git a/testbd/lfs_testbd.c b/testbd/lfs_testbd.c index f13140ed..0fae6bf4 100644 --- a/testbd/lfs_testbd.c +++ b/testbd/lfs_testbd.c @@ -199,7 +199,7 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, bd->power_cycles -= 1; if (bd->power_cycles == 0) { // sync to make sure we persist the last changes - lfs_testbd_rawsync(cfg); + assert(lfs_testbd_rawsync(cfg) == 0); // simulate power loss exit(33); } @@ -241,7 +241,7 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { bd->power_cycles -= 1; if (bd->power_cycles == 0) { // sync to make sure we persist the last changes - lfs_testbd_rawsync(cfg); + assert(lfs_testbd_rawsync(cfg) == 0); // simulate power loss exit(33); } diff --git a/tests_/test_files.toml b/tests_/test_files.toml index da132f26..565e665b 100644 --- a/tests_/test_files.toml +++ b/tests_/test_files.toml @@ -60,7 +60,7 @@ code = ''' [[case]] # rewriting files define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] -define.CHUNKSIZE = [31, 16, 1, 1025] +define.CHUNKSIZE = [31, 16, 1] code = ''' lfs_format(&lfs, &cfg) => 0; @@ -142,7 +142,7 @@ code = ''' [[case]] # appending files define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] -define.CHUNKSIZE = [31, 16, 1, 1025] +define.CHUNKSIZE = [31, 16, 1] code = ''' lfs_format(&lfs, &cfg) => 0; @@ -219,7 +219,7 @@ code = ''' [[case]] # truncating files define.SIZE1 = [32, 8192, 131072, 0, 7, 8193] define.SIZE2 = [32, 8192, 131072, 0, 7, 8193] -define.CHUNKSIZE = [31, 16, 1, 1025] +define.CHUNKSIZE = [31, 16, 1] code = ''' lfs_format(&lfs, &cfg) => 0; From aab6aa0ed939303d7788e21bb547eb0a386636fb Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 26 Jan 2020 21:37:49 -0600 Subject: [PATCH 15/41] Cleaned up test script and directory naming - Removed old tests and test scripts - Reorganize the block devices to live under one directory - Plugged new test framework into Makefile renamed: - scripts/test_.py -> scripts/test.py - tests_ -> tests - {file,ram,test}bd/* -> bd/* It took a surprising amount of effort to make the Makefile behave since it turns out the "test_%" rule could override "tests/test_%.toml.test" which is generated as part of test.py. --- Makefile | 46 +- {filebd => bd}/lfs_filebd.c | 2 +- {filebd => bd}/lfs_filebd.h | 0 {rambd => bd}/lfs_rambd.c | 2 +- {rambd => bd}/lfs_rambd.h | 0 {testbd => bd}/lfs_testbd.c | 2 +- {testbd => bd}/lfs_testbd.h | 4 +- scripts/corrupt.py | 44 -- scripts/debug.py | 112 ---- scripts/results.py | 28 - scripts/template.fmt | 96 --- scripts/test.py | 783 +++++++++++++++++++++-- scripts/test_.py | 744 --------------------- tests/test_alloc.sh | 494 -------------- {tests_ => tests}/test_alloc.toml | 0 tests/test_attrs.sh | 294 --------- {tests_ => tests}/test_attrs.toml | 0 {tests_ => tests}/test_badblocks.toml | 0 tests/test_corrupt.sh | 120 ---- tests/test_dirs.sh | 489 -------------- {tests_ => tests}/test_dirs.toml | 0 tests/test_entries.sh | 251 -------- {tests_ => tests}/test_entries.toml | 0 {tests_ => tests}/test_exhaustion.toml | 0 tests/test_files.sh | 221 ------- {tests_ => tests}/test_files.toml | 0 tests/test_format.sh | 51 -- {tests_ => tests}/test_format.toml | 0 tests/test_interspersed.sh | 190 ------ {tests_ => tests}/test_interspersed.toml | 0 tests/test_move.sh | 333 ---------- {tests_ => tests}/test_move.toml | 0 tests/test_orphan.sh | 46 -- {tests_ => tests}/test_orphans.toml | 0 tests/test_paths.sh | 202 ------ {tests_ => tests}/test_paths.toml | 0 tests/test_relocations.sh | 139 ---- {tests_ => tests}/test_relocations.toml | 0 tests/test_seek.sh | 505 --------------- {tests_ => tests}/test_seek.toml | 0 tests/test_truncate.sh | 355 ---------- {tests_ => tests}/test_truncate.toml | 0 42 files changed, 739 insertions(+), 4814 deletions(-) rename {filebd => bd}/lfs_filebd.c (99%) rename {filebd => bd}/lfs_filebd.h (100%) rename {rambd => bd}/lfs_rambd.c (99%) rename {rambd => bd}/lfs_rambd.h (100%) rename {testbd => bd}/lfs_testbd.c (99%) rename {testbd => bd}/lfs_testbd.h (98%) delete mode 100755 scripts/corrupt.py delete mode 100755 scripts/debug.py delete mode 100755 scripts/results.py delete mode 100644 scripts/template.fmt delete mode 100755 scripts/test_.py delete mode 100755 tests/test_alloc.sh rename {tests_ => tests}/test_alloc.toml (100%) delete mode 100755 tests/test_attrs.sh rename {tests_ => tests}/test_attrs.toml (100%) rename {tests_ => tests}/test_badblocks.toml (100%) delete mode 100755 tests/test_corrupt.sh delete mode 100755 tests/test_dirs.sh rename {tests_ => tests}/test_dirs.toml (100%) delete mode 100755 tests/test_entries.sh rename {tests_ => tests}/test_entries.toml (100%) rename {tests_ => tests}/test_exhaustion.toml (100%) delete mode 100755 tests/test_files.sh rename {tests_ => tests}/test_files.toml (100%) delete mode 100755 tests/test_format.sh rename {tests_ => tests}/test_format.toml (100%) delete mode 100755 tests/test_interspersed.sh rename {tests_ => tests}/test_interspersed.toml (100%) delete mode 100755 tests/test_move.sh rename {tests_ => tests}/test_move.toml (100%) delete mode 100755 tests/test_orphan.sh rename {tests_ => tests}/test_orphans.toml (100%) delete mode 100755 tests/test_paths.sh rename {tests_ => tests}/test_paths.toml (100%) delete mode 100755 tests/test_relocations.sh rename {tests_ => tests}/test_relocations.toml (100%) delete mode 100755 tests/test_seek.sh rename {tests_ => tests}/test_seek.toml (100%) delete mode 100755 tests/test_truncate.sh rename {tests_ => tests}/test_truncate.toml (100%) diff --git a/Makefile b/Makefile index 67e1c9e7..40245b8f 100644 --- a/Makefile +++ b/Makefile @@ -7,15 +7,11 @@ CC ?= gcc AR ?= ar SIZE ?= size -SRC += $(wildcard *.c rambd/*.c filebd/*.c testbd/*.c) +SRC += $(wildcard *.c bd/*.c) OBJ := $(SRC:.c=.o) DEP := $(SRC:.c=.d) ASM := $(SRC:.c=.s) -TEST := $(patsubst tests/%.sh,%,$(wildcard tests/test_*)) - -SHELL = /bin/bash -o pipefail - ifdef DEBUG override CFLAGS += -O0 -g3 else @@ -33,6 +29,10 @@ override CFLAGS += -Wextra -Wshadow -Wjump-misses-init -Wundef # Remove missing-field-initializers because of GCC bug override CFLAGS += -Wno-missing-field-initializers +ifdef VERBOSE +override TFLAGS += -v +endif + all: $(TARGET) @@ -41,38 +41,14 @@ asm: $(ASM) size: $(OBJ) $(SIZE) -t $^ -.SUFFIXES: -test: \ - test_format \ - test_dirs \ - test_files \ - test_seek \ - test_truncate \ - test_entries \ - test_interspersed \ - test_alloc \ - test_paths \ - test_attrs \ - test_move \ - test_orphan \ - test_relocations \ - test_corrupt - @rm test.c -test_%: tests/test_%.sh -ifdef QUIET - @./$< | sed -nu '/^[-=]/p' -else - ./$< -endif - -test_: - ./scripts/test_.py $(TFLAGS) +test: + ./scripts/test.py $(TFLAGS) +.SECONDEXPANSION: +test%: tests/test$$(firstword $$(subst \#, ,%)).toml + ./scripts/test.py $(TFLAGS) $@ -include $(DEP) -%?: - @echo '$($*)' - lfs: $(OBJ) $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ @@ -90,4 +66,4 @@ clean: rm -f $(OBJ) rm -f $(DEP) rm -f $(ASM) - rm -f tests_/test_*.toml.* + rm -f tests/*.toml.* diff --git a/filebd/lfs_filebd.c b/bd/lfs_filebd.c similarity index 99% rename from filebd/lfs_filebd.c rename to bd/lfs_filebd.c index a897a02d..01ba7014 100644 --- a/filebd/lfs_filebd.c +++ b/bd/lfs_filebd.c @@ -4,7 +4,7 @@ * Copyright (c) 2017, Arm Limited. All rights reserved. * SPDX-License-Identifier: BSD-3-Clause */ -#include "filebd/lfs_filebd.h" +#include "bd/lfs_filebd.h" #include #include diff --git a/filebd/lfs_filebd.h b/bd/lfs_filebd.h similarity index 100% rename from filebd/lfs_filebd.h rename to bd/lfs_filebd.h diff --git a/rambd/lfs_rambd.c b/bd/lfs_rambd.c similarity index 99% rename from rambd/lfs_rambd.c rename to bd/lfs_rambd.c index ce065563..c7c507c4 100644 --- a/rambd/lfs_rambd.c +++ b/bd/lfs_rambd.c @@ -4,7 +4,7 @@ * Copyright (c) 2017, Arm Limited. All rights reserved. * SPDX-License-Identifier: BSD-3-Clause */ -#include "rambd/lfs_rambd.h" +#include "bd/lfs_rambd.h" int lfs_rambd_createcfg(const struct lfs_config *cfg, const struct lfs_rambd_config *bdcfg) { diff --git a/rambd/lfs_rambd.h b/bd/lfs_rambd.h similarity index 100% rename from rambd/lfs_rambd.h rename to bd/lfs_rambd.h diff --git a/testbd/lfs_testbd.c b/bd/lfs_testbd.c similarity index 99% rename from testbd/lfs_testbd.c rename to bd/lfs_testbd.c index 0fae6bf4..62cba015 100644 --- a/testbd/lfs_testbd.c +++ b/bd/lfs_testbd.c @@ -5,7 +5,7 @@ * Copyright (c) 2017, Arm Limited. All rights reserved. * SPDX-License-Identifier: BSD-3-Clause */ -#include "testbd/lfs_testbd.h" +#include "bd/lfs_testbd.h" #include diff --git a/testbd/lfs_testbd.h b/bd/lfs_testbd.h similarity index 98% rename from testbd/lfs_testbd.h rename to bd/lfs_testbd.h index 9235457d..78580d44 100644 --- a/testbd/lfs_testbd.h +++ b/bd/lfs_testbd.h @@ -10,8 +10,8 @@ #include "lfs.h" #include "lfs_util.h" -#include "rambd/lfs_rambd.h" -#include "filebd/lfs_filebd.h" +#include "bd/lfs_rambd.h" +#include "bd/lfs_filebd.h" #ifdef __cplusplus extern "C" diff --git a/scripts/corrupt.py b/scripts/corrupt.py deleted file mode 100755 index c452c423..00000000 --- a/scripts/corrupt.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python2 - -import struct -import sys -import os -import argparse - -def corrupt(block): - with open(block, 'r+b') as file: - # skip rev - file.read(4) - - # go to last commit - tag = 0xffffffff - while True: - try: - ntag, = struct.unpack('>I', file.read(4)) - except struct.error: - break - - tag ^= ntag - size = (tag & 0x3ff) if (tag & 0x3ff) != 0x3ff else 0 - file.seek(size, os.SEEK_CUR) - - # lob off last 3 bytes - file.seek(-(size + 3), os.SEEK_CUR) - file.truncate() - -def main(args): - if args.n or not args.blocks: - with open('blocks/.history', 'rb') as file: - for i in range(int(args.n or 1)): - last, = struct.unpack('I', data) - except struct.error: - break - - tag ^= ntag - off += 4 - - type = (tag & 0x7ff00000) >> 20 - id = (tag & 0x000ffc00) >> 10 - size = (tag & 0x000003ff) >> 0 - iscrc = (type & 0x700) == 0x500 - - data = file.read(size if size != 0x3ff else 0) - if iscrc: - crc = binascii.crc32(data[:4], crc) - else: - crc = binascii.crc32(data, crc) - - print '%04x: %08x %-15s %3s %4s %-23s %-8s' % ( - off, tag, - typeof(type) + (' bad!' if iscrc and ~crc else ''), - hex(id)[2:] if id != 0x3ff else '.', - size if size != 0x3ff else 'x', - ' '.join('%02x' % ord(c) for c in data[:8]), - ''.join(c if c >= ' ' and c <= '~' else '.' for c in data[:8])) - - off += size if size != 0x3ff else 0 - if iscrc: - crc = 0 - tag ^= (type & 1) << 31 - - return 0 - -if __name__ == "__main__": - import sys - sys.exit(main(*sys.argv[1:])) diff --git a/scripts/results.py b/scripts/results.py deleted file mode 100755 index 0e5cfc63..00000000 --- a/scripts/results.py +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env python2 - -import struct -import sys -import time -import os -import re - -def main(): - with open('blocks/.config') as file: - read_size, prog_size, block_size, block_count = ( - struct.unpack(' -#include -#include - - -// test stuff -static void test_assert(const char *file, unsigned line, - const char *s, uintmax_t v, uintmax_t e) {{ - if (v != e) {{ - fprintf(stderr, "\033[97m%s:%u: \033[91m" - "assert failed with %jd, expected %jd\033[0m\n" - " %s\n\n", file, line, v, e, s); - exit(-2); - }} -}} - -#define test_assert(v, e) \ - test_assert(__FILE__, __LINE__, #v " => " #e, v, e) - -// implicit variable for asserts -uintmax_t test; - -// utility functions for traversals -static int __attribute__((used)) test_count(void *p, lfs_block_t b) {{ - (void)b; - unsigned *u = (unsigned*)p; - *u += 1; - return 0; -}} - -// lfs declarations -lfs_t lfs; -lfs_emubd_t bd; -// other declarations for convenience -lfs_file_t file; -lfs_dir_t dir; -struct lfs_info info; -uint8_t buffer[1024]; -char path[1024]; - -// test configuration options -#ifndef LFS_READ_SIZE -#define LFS_READ_SIZE 16 -#endif - -#ifndef LFS_PROG_SIZE -#define LFS_PROG_SIZE LFS_READ_SIZE -#endif - -#ifndef LFS_BLOCK_SIZE -#define LFS_BLOCK_SIZE 512 -#endif - -#ifndef LFS_BLOCK_COUNT -#define LFS_BLOCK_COUNT 1024 -#endif - -#ifndef LFS_BLOCK_CYCLES -#define LFS_BLOCK_CYCLES 1024 -#endif - -#ifndef LFS_CACHE_SIZE -#define LFS_CACHE_SIZE (64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE) -#endif - -#ifndef LFS_LOOKAHEAD_SIZE -#define LFS_LOOKAHEAD_SIZE 16 -#endif - -const struct lfs_config cfg = {{ - .context = &bd, - .read = &lfs_emubd_read, - .prog = &lfs_emubd_prog, - .erase = &lfs_emubd_erase, - .sync = &lfs_emubd_sync, - - .read_size = LFS_READ_SIZE, - .prog_size = LFS_PROG_SIZE, - .block_size = LFS_BLOCK_SIZE, - .block_count = LFS_BLOCK_COUNT, - .block_cycles = LFS_BLOCK_CYCLES, - .cache_size = LFS_CACHE_SIZE, - .lookahead_size = LFS_LOOKAHEAD_SIZE, -}}; - - -// Entry point -int main(void) {{ - lfs_emubd_create(&cfg, "blocks"); - -{tests} - lfs_emubd_destroy(&cfg); -}} diff --git a/scripts/test.py b/scripts/test.py index 3135c654..02801f8c 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -1,81 +1,744 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 +# This script manages littlefs tests, which are configured with +# .toml files stored in the tests directory. +# + +import toml +import glob import re -import sys -import subprocess import os +import io +import itertools as it +import collections.abc as abc +import subprocess as sp +import base64 +import sys +import copy +import shlex +import pty +import errno +import signal + +TESTDIR = 'tests' +RULES = """ +define FLATTEN +tests/%$(subst /,.,$(target)): $(target) + ./scripts/explode_asserts.py $$< -o $$@ +endef +$(foreach target,$(SRC),$(eval $(FLATTEN))) + +-include tests/*.d +.SECONDARY: +%.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.$f) + $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ +""" +GLOBALS = """ +//////////////// AUTOGENERATED TEST //////////////// +#include "lfs.h" +#include "bd/lfs_testbd.h" +#include +extern const char *lfs_testbd_path; +extern uint32_t lfs_testbd_cycles; +""" +DEFINES = { + 'LFS_READ_SIZE': 16, + 'LFS_PROG_SIZE': 'LFS_READ_SIZE', + 'LFS_BLOCK_SIZE': 512, + 'LFS_BLOCK_COUNT': 1024, + 'LFS_BLOCK_CYCLES': -1, + 'LFS_CACHE_SIZE': '(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)', + 'LFS_LOOKAHEAD_SIZE': 16, + 'LFS_ERASE_VALUE': 0xff, + 'LFS_ERASE_CYCLES': 0, + 'LFS_BADBLOCK_BEHAVIOR': 'LFS_TESTBD_BADBLOCK_NOPROG', +} +PROLOGUE = """ + // prologue + __attribute__((unused)) lfs_t lfs; + __attribute__((unused)) lfs_testbd_t bd; + __attribute__((unused)) lfs_file_t file; + __attribute__((unused)) lfs_dir_t dir; + __attribute__((unused)) struct lfs_info info; + __attribute__((unused)) char path[1024]; + __attribute__((unused)) uint8_t buffer[1024]; + __attribute__((unused)) lfs_size_t size; + __attribute__((unused)) int err; + + __attribute__((unused)) const struct lfs_config cfg = { + .context = &bd, + .read = lfs_testbd_read, + .prog = lfs_testbd_prog, + .erase = lfs_testbd_erase, + .sync = lfs_testbd_sync, + .read_size = LFS_READ_SIZE, + .prog_size = LFS_PROG_SIZE, + .block_size = LFS_BLOCK_SIZE, + .block_count = LFS_BLOCK_COUNT, + .block_cycles = LFS_BLOCK_CYCLES, + .cache_size = LFS_CACHE_SIZE, + .lookahead_size = LFS_LOOKAHEAD_SIZE, + }; -def generate(test): - with open("scripts/template.fmt") as file: - template = file.read() + __attribute__((unused)) const struct lfs_testbd_config bdcfg = { + .erase_value = LFS_ERASE_VALUE, + .erase_cycles = LFS_ERASE_CYCLES, + .badblock_behavior = LFS_BADBLOCK_BEHAVIOR, + .power_cycles = lfs_testbd_cycles, + }; - haslines = 'TEST_LINE' in os.environ and 'TEST_FILE' in os.environ + lfs_testbd_createcfg(&cfg, lfs_testbd_path, &bdcfg) => 0; +""" +EPILOGUE = """ + // epilogue + lfs_testbd_destroy(&cfg) => 0; +""" +PASS = '\033[32m✓\033[0m' +FAIL = '\033[31m✗\033[0m' - lines = [] - for offset, line in enumerate( - re.split('(?<=(?:.;| [{}]))\n', test.read())): - match = re.match('((?: *\n)*)( *)(.*)=>(.*);', - line, re.DOTALL | re.MULTILINE) - if match: - preface, tab, test, expect = match.groups() - lines.extend(['']*preface.count('\n')) - lines.append(tab+'test_assert({test}, {expect});'.format( - test=test.strip(), expect=expect.strip())) +class TestFailure(Exception): + def __init__(self, case, returncode=None, stdout=None, assert_=None): + self.case = case + self.returncode = returncode + self.stdout = stdout + self.assert_ = assert_ + +class TestCase: + def __init__(self, config, filter=filter, + suite=None, caseno=None, lineno=None, **_): + self.config = config + self.filter = filter + self.suite = suite + self.caseno = caseno + self.lineno = lineno + + self.code = config['code'] + self.code_lineno = config['code_lineno'] + self.defines = config.get('define', {}) + self.if_ = config.get('if', None) + self.in_ = config.get('in', None) + + def __str__(self): + if hasattr(self, 'permno'): + if any(k not in self.case.defines for k in self.defines): + return '%s#%d#%d (%s)' % ( + self.suite.name, self.caseno, self.permno, ', '.join( + '%s=%s' % (k, v) for k, v in self.defines.items() + if k not in self.case.defines)) + else: + return '%s#%d#%d' % ( + self.suite.name, self.caseno, self.permno) else: - lines.append(line) + return '%s#%d' % ( + self.suite.name, self.caseno) - # Create test file - with open('test.c', 'w') as file: - if 'TEST_LINE' in os.environ and 'TEST_FILE' in os.environ: - lines.insert(0, '#line %d "%s"' % ( - int(os.environ['TEST_LINE']) + 1, - os.environ['TEST_FILE'])) - lines.append('#line %d "test.c"' % ( - template[:template.find('{tests}')].count('\n') - + len(lines) + 2)) + def permute(self, class_=None, defines={}, permno=None, **_): + ncase = (class_ or type(self))(self.config) + for k, v in self.__dict__.items(): + setattr(ncase, k, v) + ncase.case = self + ncase.perms = [ncase] + ncase.permno = permno + ncase.defines = defines + return ncase - file.write(template.format(tests='\n'.join(lines))) + def build(self, f, **_): + # prologue + for k, v in sorted(self.defines.items()): + if k not in self.suite.defines: + f.write('#define %s %s\n' % (k, v)) - # Remove build artifacts to force rebuild - try: - os.remove('test.o') - os.remove('lfs') - except OSError: - pass + f.write('void test_case%d(%s) {' % (self.caseno, ','.join( + '\n'+8*' '+'__attribute__((unused)) intmax_t %s' % k + for k in sorted(self.perms[0].defines) + if k not in self.defines))) -def compile(): - subprocess.check_call([ - os.environ.get('MAKE', 'make'), - '--no-print-directory', '-s']) + f.write(PROLOGUE) + f.write('\n') + f.write(4*' '+'// test case %d\n' % self.caseno) + f.write(4*' '+'#line %d "%s"\n' % (self.code_lineno, self.suite.path)) -def execute(): - if 'EXEC' in os.environ: - subprocess.check_call([os.environ['EXEC'], "./lfs"]) - else: - subprocess.check_call(["./lfs"]) + # test case goes here + f.write(self.code) -def main(test=None): - try: - if test and not test.startswith('-'): - with open(test) as file: - generate(file) + # epilogue + f.write(EPILOGUE) + f.write('}\n') + + for k, v in sorted(self.defines.items()): + if k not in self.suite.defines: + f.write('#undef %s\n' % k) + + def shouldtest(self, **args): + if (self.filter is not None and + len(self.filter) >= 1 and + self.filter[0] != self.caseno): + return False + elif (self.filter is not None and + len(self.filter) >= 2 and + self.filter[1] != self.permno): + return False + elif args.get('no_internal', False) and self.in_ is not None: + return False + elif self.if_ is not None: + return eval(self.if_, None, self.defines.copy()) + else: + return True + + def test(self, exec=[], persist=False, cycles=None, + gdb=False, failure=None, **args): + # build command + cmd = exec + ['./%s.test' % self.suite.path, + repr(self.caseno), repr(self.permno)] + + # persist disk or keep in RAM for speed? + if persist: + if persist != 'noerase': + try: + os.remove(self.suite.path + '.disk') + if args.get('verbose', False): + print('rm', self.suite.path + '.disk') + except FileNotFoundError: + pass + + cmd.append(self.suite.path + '.disk') + + # simulate power-loss after n cycles? + if cycles: + cmd.append(str(cycles)) + + # failed? drop into debugger? + if gdb and failure: + ncmd = ['gdb'] + if gdb == 'assert': + ncmd.extend(['-ex', 'r']) + if failure.assert_: + ncmd.extend(['-ex', 'up 2']) + elif gdb == 'start': + ncmd.extend([ + '-ex', 'b %s:%d' % (self.suite.path, self.code_lineno), + '-ex', 'r']) + ncmd.extend(['--args'] + cmd) + + if args.get('verbose', False): + print(' '.join(shlex.quote(c) for c in ncmd)) + signal.signal(signal.SIGINT, signal.SIG_IGN) + sys.exit(sp.call(ncmd)) + + # run test case! + mpty, spty = pty.openpty() + if args.get('verbose', False): + print(' '.join(shlex.quote(c) for c in cmd)) + proc = sp.Popen(cmd, stdout=spty, stderr=spty) + os.close(spty) + mpty = os.fdopen(mpty, 'r', 1) + stdout = [] + assert_ = None + while True: + try: + line = mpty.readline() + except OSError as e: + if e.errno == errno.EIO: + break + raise + stdout.append(line) + if args.get('verbose', False): + sys.stdout.write(line) + # intercept asserts + m = re.match( + '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' + .format('(?:\033\[[\d;]*.| )*', 'assert'), + line) + if m and assert_ is None: + try: + with open(m.group(1)) as f: + lineno = int(m.group(2)) + line = next(it.islice(f, lineno-1, None)).strip('\n') + assert_ = { + 'path': m.group(1), + 'line': line, + 'lineno': lineno, + 'message': m.group(3)} + except: + pass + proc.wait() + + # did we pass? + if proc.returncode != 0: + raise TestFailure(self, proc.returncode, stdout, assert_) else: - generate(sys.stdin) + return PASS + +class ValgrindTestCase(TestCase): + def __init__(self, config, **args): + self.leaky = config.get('leaky', False) + super().__init__(config, **args) + + def shouldtest(self, **args): + return not self.leaky and super().shouldtest(**args) + + def test(self, exec=[], **args): + exec = exec + [ + 'valgrind', + '--leak-check=full', + '--error-exitcode=4', + '-q'] + return super().test(exec=exec, **args) + +class ReentrantTestCase(TestCase): + def __init__(self, config, **args): + self.reentrant = config.get('reentrant', False) + super().__init__(config, **args) + + def shouldtest(self, **args): + return self.reentrant and super().shouldtest(**args) + + def test(self, exec=[], persist=False, gdb=False, failure=None, **args): + for cycles in it.count(1): + # clear disk first? + if cycles == 1 and persist != 'noerase': + persist = 'erase' + else: + persist = 'noerase' + + # exact cycle we should drop into debugger? + if gdb and failure and failure.cycleno == cycles: + return super().test(gdb=gdb, persist=persist, cycles=cycles, + failure=failure, **args) + + # run tests, but kill the program after prog/erase has + # been hit n cycles. We exit with a special return code if the + # program has not finished, since this isn't a test failure. + try: + return super().test(persist=persist, cycles=cycles, **args) + except TestFailure as nfailure: + if nfailure.returncode == 33: + continue + else: + nfailure.cycleno = cycles + raise + +class TestSuite: + def __init__(self, path, classes=[TestCase], defines={}, + filter=None, **args): + self.name = os.path.basename(path) + if self.name.endswith('.toml'): + self.name = self.name[:-len('.toml')] + self.path = path + self.classes = classes + self.defines = defines.copy() + self.filter = filter + + with open(path) as f: + # load tests + config = toml.load(f) + + # find line numbers + f.seek(0) + linenos = [] + code_linenos = [] + for i, line in enumerate(f): + if re.match(r'\[\[\s*case\s*\]\]', line): + linenos.append(i+1) + if re.match(r'code\s*=\s*(\'\'\'|""")', line): + code_linenos.append(i+2) + + code_linenos.reverse() + + # grab global config + for k, v in config.get('define', {}).items(): + if k not in self.defines: + self.defines[k] = v + self.code = config.get('code', None) + if self.code is not None: + self.code_lineno = code_linenos.pop() + + # create initial test cases + self.cases = [] + for i, (case, lineno) in enumerate(zip(config['case'], linenos)): + # code lineno? + if 'code' in case: + case['code_lineno'] = code_linenos.pop() + # give our case's config a copy of our "global" config + for k, v in config.items(): + if k not in case: + case[k] = v + # initialize test case + self.cases.append(TestCase(case, filter=filter, + suite=self, caseno=i+1, lineno=lineno, **args)) + + def __str__(self): + return self.name + + def __lt__(self, other): + return self.name < other.name + + def permute(self, **args): + for case in self.cases: + # lets find all parameterized definitions, in one of [args.D, + # suite.defines, case.defines, DEFINES]. Note that each of these + # can be either a dict of defines, or a list of dicts, expressing + # an initial set of permutations. + pending = [{}] + for inits in [self.defines, case.defines, DEFINES]: + if not isinstance(inits, list): + inits = [inits] + + npending = [] + for init, pinit in it.product(inits, pending): + ninit = pinit.copy() + for k, v in init.items(): + if k not in ninit: + try: + ninit[k] = eval(v) + except: + ninit[k] = v + npending.append(ninit) + + pending = npending + + # expand permutations + pending = list(reversed(pending)) + expanded = [] + while pending: + perm = pending.pop() + for k, v in sorted(perm.items()): + if not isinstance(v, str) and isinstance(v, abc.Iterable): + for nv in reversed(v): + nperm = perm.copy() + nperm[k] = nv + pending.append(nperm) + break + else: + expanded.append(perm) + + # generate permutations + case.perms = [] + for i, (class_, defines) in enumerate( + it.product(self.classes, expanded)): + case.perms.append(case.permute( + class_, defines, permno=i+1, **args)) + + # also track non-unique defines + case.defines = {} + for k, v in case.perms[0].defines.items(): + if all(perm.defines[k] == v for perm in case.perms): + case.defines[k] = v + + # track all perms and non-unique defines + self.perms = [] + for case in self.cases: + self.perms.extend(case.perms) + + self.defines = {} + for k, v in self.perms[0].defines.items(): + if all(perm.defines.get(k, None) == v for perm in self.perms): + self.defines[k] = v + + return self.perms + + def build(self, **args): + # build test files + tf = open(self.path + '.test.c.t', 'w') + tf.write(GLOBALS) + if self.code is not None: + tf.write('#line %d "%s"\n' % (self.code_lineno, self.path)) + tf.write(self.code) + + tfs = {None: tf} + for case in self.cases: + if case.in_ not in tfs: + tfs[case.in_] = open(self.path+'.'+ + case.in_.replace('/', '.')+'.t', 'w') + tfs[case.in_].write('#line 1 "%s"\n' % case.in_) + with open(case.in_) as f: + for line in f: + tfs[case.in_].write(line) + tfs[case.in_].write('\n') + tfs[case.in_].write(GLOBALS) + + tfs[case.in_].write('\n') + case.build(tfs[case.in_], **args) + + tf.write('\n') + tf.write('const char *lfs_testbd_path;\n') + tf.write('uint32_t lfs_testbd_cycles;\n') + tf.write('int main(int argc, char **argv) {\n') + tf.write(4*' '+'int case_ = (argc > 1) ? atoi(argv[1]) : 0;\n') + tf.write(4*' '+'int perm = (argc > 2) ? atoi(argv[2]) : 0;\n') + tf.write(4*' '+'lfs_testbd_path = (argc > 3) ? argv[3] : NULL;\n') + tf.write(4*' '+'lfs_testbd_cycles = (argc > 4) ? atoi(argv[4]) : 0;\n') + for perm in self.perms: + # test declaration + tf.write(4*' '+'extern void test_case%d(%s);\n' % ( + perm.caseno, ', '.join( + 'intmax_t %s' % k for k in sorted(perm.defines) + if k not in perm.case.defines))) + # test call + tf.write(4*' '+ + 'if (argc < 3 || (case_ == %d && perm == %d)) {' + ' test_case%d(%s); ' + '}\n' % (perm.caseno, perm.permno, perm.caseno, ', '.join( + str(v) for k, v in sorted(perm.defines.items()) + if k not in perm.case.defines))) + tf.write('}\n') + + for tf in tfs.values(): + tf.close() + + # write makefiles + with open(self.path + '.mk', 'w') as mk: + mk.write(RULES.replace(4*' ', '\t')) + mk.write('\n') + + # add truely global defines globally + for k, v in sorted(self.defines.items()): + mk.write('%s: override CFLAGS += -D%s=%r\n' % ( + self.path+'.test', k, v)) + + for path in tfs: + if path is None: + mk.write('%s: %s | %s\n' % ( + self.path+'.test.c', + self.path, + self.path+'.test.c.t')) + else: + mk.write('%s: %s %s | %s\n' % ( + self.path+'.'+path.replace('/', '.'), + self.path, path, + self.path+'.'+path.replace('/', '.')+'.t')) + mk.write('\t./scripts/explode_asserts.py $| -o $@\n') + + self.makefile = self.path + '.mk' + self.target = self.path + '.test' + return self.makefile, self.target + + def test(self, **args): + # run test suite! + if not args.get('verbose', True): + sys.stdout.write(self.name + ' ') + sys.stdout.flush() + for perm in self.perms: + if not perm.shouldtest(**args): + continue + + try: + result = perm.test(**args) + except TestFailure as failure: + perm.result = failure + if not args.get('verbose', True): + sys.stdout.write(FAIL) + sys.stdout.flush() + if not args.get('keep_going', False): + if not args.get('verbose', True): + sys.stdout.write('\n') + raise + else: + perm.result = PASS + if not args.get('verbose', True): + sys.stdout.write(PASS) + sys.stdout.flush() + + if not args.get('verbose', True): + sys.stdout.write('\n') + +def main(**args): + # figure out explicit defines + defines = {} + for define in args['D']: + k, v, *_ = define.split('=', 2) + [''] + defines[k] = v + + # and what class of TestCase to run + classes = [] + if args.get('normal', False): + classes.append(TestCase) + if args.get('reentrant', False): + classes.append(ReentrantTestCase) + if args.get('valgrind', False): + classes.append(ValgrindTestCase) + if not classes: + classes = [TestCase] + + suites = [] + for testpath in args['testpaths']: + # optionally specified test case/perm + testpath, *filter = testpath.split('#') + filter = [int(f) for f in filter] + + # figure out the suite's toml file + if os.path.isdir(testpath): + testpath = testpath + '/test_*.toml' + elif os.path.isfile(testpath): + testpath = testpath + elif testpath.endswith('.toml'): + testpath = TESTDIR + '/' + testpath + else: + testpath = TESTDIR + '/' + testpath + '.toml' + + # find tests + for path in glob.glob(testpath): + suites.append(TestSuite(path, classes, defines, filter, **args)) + + # sort for reproducability + suites = sorted(suites) + + # generate permutations + for suite in suites: + suite.permute(**args) + + # build tests in parallel + print('====== building ======') + makefiles = [] + targets = [] + for suite in suites: + makefile, target = suite.build(**args) + makefiles.append(makefile) + targets.append(target) + + cmd = (['make', '-f', 'Makefile'] + + list(it.chain.from_iterable(['-f', m] for m in makefiles)) + + [target for target in targets]) + mpty, spty = pty.openpty() + if args.get('verbose', False): + print(' '.join(shlex.quote(c) for c in cmd)) + proc = sp.Popen(cmd, stdout=spty, stderr=spty) + os.close(spty) + mpty = os.fdopen(mpty, 'r', 1) + stdout = [] + while True: + try: + line = mpty.readline() + except OSError as e: + if e.errno == errno.EIO: + break + raise + stdout.append(line) + if args.get('verbose', False): + sys.stdout.write(line) + # intercept warnings + m = re.match( + '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' + .format('(?:\033\[[\d;]*.| )*', 'warning'), + line) + if m and not args.get('verbose', False): + try: + with open(m.group(1)) as f: + lineno = int(m.group(2)) + line = next(it.islice(f, lineno-1, None)).strip('\n') + sys.stdout.write( + "\033[01m{path}:{lineno}:\033[01;35mwarning:\033[m " + "{message}\n{line}\n\n".format( + path=m.group(1), line=line, lineno=lineno, + message=m.group(3))) + except: + pass + proc.wait() + + if proc.returncode != 0: + if not args.get('verbose', False): + for line in stdout: + sys.stdout.write(line) + sys.exit(-3) + + print('built %d test suites, %d test cases, %d permutations' % ( + len(suites), + sum(len(suite.cases) for suite in suites), + sum(len(suite.perms) for suite in suites))) + + filtered = 0 + for suite in suites: + for perm in suite.perms: + filtered += perm.shouldtest(**args) + if filtered != sum(len(suite.perms) for suite in suites): + print('filtered down to %d permutations' % filtered) + + print('====== testing ======') + try: + for suite in suites: + suite.test(**args) + except TestFailure: + pass - compile() + print('====== results ======') + passed = 0 + failed = 0 + for suite in suites: + for perm in suite.perms: + if not hasattr(perm, 'result'): + continue - if test == '-s': - sys.exit(1) + if perm.result == PASS: + passed += 1 + else: + sys.stdout.write( + "\033[01m{path}:{lineno}:\033[01;31mfailure:\033[m " + "{perm} failed with {returncode}\n".format( + perm=perm, path=perm.suite.path, lineno=perm.lineno, + returncode=perm.result.returncode or 0)) + if perm.result.stdout: + for line in (perm.result.stdout + if not perm.result.assert_ + else perm.result.stdout[:-1]): + sys.stdout.write(line) + if perm.result.assert_: + sys.stdout.write( + "\033[01m{path}:{lineno}:\033[01;31massert:\033[m " + "{message}\n{line}\n".format( + **perm.result.assert_)) + else: + for line in perm.result.stdout: + sys.stdout.write(line) + sys.stdout.write('\n') + failed += 1 - execute() + if args.get('gdb', False): + failure = None + for suite in suites: + for perm in suite.perms: + if getattr(perm, 'result', PASS) != PASS: + failure = perm.result + if failure is not None: + print('======= gdb ======') + # drop into gdb + failure.case.test(failure=failure, **args) + sys.exit(0) - except subprocess.CalledProcessError: - # Python stack trace is counterproductive, just exit - sys.exit(2) - except KeyboardInterrupt: - # Python stack trace is counterproductive, just exit - sys.exit(3) + print('tests passed: %d' % passed) + print('tests failed: %d' % failed) + return 1 if failed > 0 else 0 if __name__ == "__main__": - main(*sys.argv[1:]) + import argparse + parser = argparse.ArgumentParser( + description="Run parameterized tests in various configurations.") + parser.add_argument('testpaths', nargs='*', default=[TESTDIR], + help="Description of test(s) to run. By default, this is all tests \ + found in the \"{0}\" directory. Here, you can specify a different \ + directory of tests, a specific file, a suite by name, and even a \ + specific test case by adding brackets. For example \ + \"test_dirs[0]\" or \"{0}/test_dirs.toml[0]\".".format(TESTDIR)) + parser.add_argument('-D', action='append', default=[], + help="Overriding parameter definitions.") + parser.add_argument('-v', '--verbose', action='store_true', + help="Output everything that is happening.") + parser.add_argument('-k', '--keep-going', action='store_true', + help="Run all tests instead of stopping on first error. Useful for CI.") + parser.add_argument('-p', '--persist', choices=['erase', 'noerase'], + nargs='?', const='erase', + help="Store disk image in a file.") + parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], + nargs='?', const='assert', + help="Drop into gdb on test failure.") + parser.add_argument('--no-internal', action='store_true', + help="Don't run tests that require internal knowledge.") + parser.add_argument('-n', '--normal', action='store_true', + help="Run tests normally.") + parser.add_argument('-r', '--reentrant', action='store_true', + help="Run reentrant tests with simulated power-loss.") + parser.add_argument('-V', '--valgrind', action='store_true', + help="Run non-leaky tests under valgrind to check for memory leaks.") + parser.add_argument('-e', '--exec', default=[], type=lambda e: e.split(' '), + help="Run tests with another executable prefixed on the command line.") + sys.exit(main(**vars(parser.parse_args()))) diff --git a/scripts/test_.py b/scripts/test_.py deleted file mode 100755 index 4593b40a..00000000 --- a/scripts/test_.py +++ /dev/null @@ -1,744 +0,0 @@ -#!/usr/bin/env python3 - -# This script manages littlefs tests, which are configured with -# .toml files stored in the tests directory. -# - -import toml -import glob -import re -import os -import io -import itertools as it -import collections.abc as abc -import subprocess as sp -import base64 -import sys -import copy -import shlex -import pty -import errno -import signal - -TESTDIR = 'tests_' -RULES = """ -define FLATTEN -tests_/%$(subst /,.,$(target)): $(target) - ./scripts/explode_asserts.py $$< -o $$@ -endef -$(foreach target,$(SRC),$(eval $(FLATTEN))) - --include tests_/*.d - -.SECONDARY: -%.test: %.test.o $(foreach f,$(subst /,.,$(SRC:.c=.o)),%.$f) - $(CC) $(CFLAGS) $^ $(LFLAGS) -o $@ -""" -GLOBALS = """ -//////////////// AUTOGENERATED TEST //////////////// -#include "lfs.h" -#include "testbd/lfs_testbd.h" -#include -extern const char *lfs_testbd_path; -extern uint32_t lfs_testbd_cycles; -""" -DEFINES = { - 'LFS_READ_SIZE': 16, - 'LFS_PROG_SIZE': 'LFS_READ_SIZE', - 'LFS_BLOCK_SIZE': 512, - 'LFS_BLOCK_COUNT': 1024, - 'LFS_BLOCK_CYCLES': -1, - 'LFS_CACHE_SIZE': '(64 % LFS_PROG_SIZE == 0 ? 64 : LFS_PROG_SIZE)', - 'LFS_LOOKAHEAD_SIZE': 16, - 'LFS_ERASE_VALUE': 0xff, - 'LFS_ERASE_CYCLES': 0, - 'LFS_BADBLOCK_BEHAVIOR': 'LFS_TESTBD_BADBLOCK_NOPROG', -} -PROLOGUE = """ - // prologue - __attribute__((unused)) lfs_t lfs; - __attribute__((unused)) lfs_testbd_t bd; - __attribute__((unused)) lfs_file_t file; - __attribute__((unused)) lfs_dir_t dir; - __attribute__((unused)) struct lfs_info info; - __attribute__((unused)) char path[1024]; - __attribute__((unused)) uint8_t buffer[1024]; - __attribute__((unused)) lfs_size_t size; - __attribute__((unused)) int err; - - __attribute__((unused)) const struct lfs_config cfg = { - .context = &bd, - .read = lfs_testbd_read, - .prog = lfs_testbd_prog, - .erase = lfs_testbd_erase, - .sync = lfs_testbd_sync, - .read_size = LFS_READ_SIZE, - .prog_size = LFS_PROG_SIZE, - .block_size = LFS_BLOCK_SIZE, - .block_count = LFS_BLOCK_COUNT, - .block_cycles = LFS_BLOCK_CYCLES, - .cache_size = LFS_CACHE_SIZE, - .lookahead_size = LFS_LOOKAHEAD_SIZE, - }; - - __attribute__((unused)) const struct lfs_testbd_config bdcfg = { - .erase_value = LFS_ERASE_VALUE, - .erase_cycles = LFS_ERASE_CYCLES, - .badblock_behavior = LFS_BADBLOCK_BEHAVIOR, - .power_cycles = lfs_testbd_cycles, - }; - - lfs_testbd_createcfg(&cfg, lfs_testbd_path, &bdcfg) => 0; -""" -EPILOGUE = """ - // epilogue - lfs_testbd_destroy(&cfg) => 0; -""" -PASS = '\033[32m✓\033[0m' -FAIL = '\033[31m✗\033[0m' - -class TestFailure(Exception): - def __init__(self, case, returncode=None, stdout=None, assert_=None): - self.case = case - self.returncode = returncode - self.stdout = stdout - self.assert_ = assert_ - -class TestCase: - def __init__(self, config, filter=filter, - suite=None, caseno=None, lineno=None, **_): - self.config = config - self.filter = filter - self.suite = suite - self.caseno = caseno - self.lineno = lineno - - self.code = config['code'] - self.code_lineno = config['code_lineno'] - self.defines = config.get('define', {}) - self.if_ = config.get('if', None) - self.in_ = config.get('in', None) - - def __str__(self): - if hasattr(self, 'permno'): - if any(k not in self.case.defines for k in self.defines): - return '%s#%d#%d (%s)' % ( - self.suite.name, self.caseno, self.permno, ', '.join( - '%s=%s' % (k, v) for k, v in self.defines.items() - if k not in self.case.defines)) - else: - return '%s#%d#%d' % ( - self.suite.name, self.caseno, self.permno) - else: - return '%s#%d' % ( - self.suite.name, self.caseno) - - def permute(self, class_=None, defines={}, permno=None, **_): - ncase = (class_ or type(self))(self.config) - for k, v in self.__dict__.items(): - setattr(ncase, k, v) - ncase.case = self - ncase.perms = [ncase] - ncase.permno = permno - ncase.defines = defines - return ncase - - def build(self, f, **_): - # prologue - for k, v in sorted(self.defines.items()): - if k not in self.suite.defines: - f.write('#define %s %s\n' % (k, v)) - - f.write('void test_case%d(%s) {' % (self.caseno, ','.join( - '\n'+8*' '+'__attribute__((unused)) intmax_t %s' % k - for k in sorted(self.perms[0].defines) - if k not in self.defines))) - - f.write(PROLOGUE) - f.write('\n') - f.write(4*' '+'// test case %d\n' % self.caseno) - f.write(4*' '+'#line %d "%s"\n' % (self.code_lineno, self.suite.path)) - - # test case goes here - f.write(self.code) - - # epilogue - f.write(EPILOGUE) - f.write('}\n') - - for k, v in sorted(self.defines.items()): - if k not in self.suite.defines: - f.write('#undef %s\n' % k) - - def shouldtest(self, **args): - if (self.filter is not None and - len(self.filter) >= 1 and - self.filter[0] != self.caseno): - return False - elif (self.filter is not None and - len(self.filter) >= 2 and - self.filter[1] != self.permno): - return False - elif args.get('no_internal', False) and self.in_ is not None: - return False - elif self.if_ is not None: - return eval(self.if_, None, self.defines.copy()) - else: - return True - - def test(self, exec=[], persist=False, cycles=None, - gdb=False, failure=None, **args): - # build command - cmd = exec + ['./%s.test' % self.suite.path, - repr(self.caseno), repr(self.permno)] - - # persist disk or keep in RAM for speed? - if persist: - if persist != 'noerase': - try: - os.remove(self.suite.path + '.disk') - if args.get('verbose', False): - print('rm', self.suite.path + '.disk') - except FileNotFoundError: - pass - - cmd.append(self.suite.path + '.disk') - - # simulate power-loss after n cycles? - if cycles: - cmd.append(str(cycles)) - - # failed? drop into debugger? - if gdb and failure: - ncmd = ['gdb'] - if gdb == 'assert': - ncmd.extend(['-ex', 'r']) - if failure.assert_: - ncmd.extend(['-ex', 'up 2']) - elif gdb == 'start': - ncmd.extend([ - '-ex', 'b %s:%d' % (self.suite.path, self.code_lineno), - '-ex', 'r']) - ncmd.extend(['--args'] + cmd) - - if args.get('verbose', False): - print(' '.join(shlex.quote(c) for c in ncmd)) - signal.signal(signal.SIGINT, signal.SIG_IGN) - sys.exit(sp.call(ncmd)) - - # run test case! - mpty, spty = pty.openpty() - if args.get('verbose', False): - print(' '.join(shlex.quote(c) for c in cmd)) - proc = sp.Popen(cmd, stdout=spty, stderr=spty) - os.close(spty) - mpty = os.fdopen(mpty, 'r', 1) - stdout = [] - assert_ = None - while True: - try: - line = mpty.readline() - except OSError as e: - if e.errno == errno.EIO: - break - raise - stdout.append(line) - if args.get('verbose', False): - sys.stdout.write(line) - # intercept asserts - m = re.match( - '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' - .format('(?:\033\[[\d;]*.| )*', 'assert'), - line) - if m and assert_ is None: - try: - with open(m.group(1)) as f: - lineno = int(m.group(2)) - line = next(it.islice(f, lineno-1, None)).strip('\n') - assert_ = { - 'path': m.group(1), - 'line': line, - 'lineno': lineno, - 'message': m.group(3)} - except: - pass - proc.wait() - - # did we pass? - if proc.returncode != 0: - raise TestFailure(self, proc.returncode, stdout, assert_) - else: - return PASS - -class ValgrindTestCase(TestCase): - def __init__(self, config, **args): - self.leaky = config.get('leaky', False) - super().__init__(config, **args) - - def shouldtest(self, **args): - return not self.leaky and super().shouldtest(**args) - - def test(self, exec=[], **args): - exec = exec + [ - 'valgrind', - '--leak-check=full', - '--error-exitcode=4', - '-q'] - return super().test(exec=exec, **args) - -class ReentrantTestCase(TestCase): - def __init__(self, config, **args): - self.reentrant = config.get('reentrant', False) - super().__init__(config, **args) - - def shouldtest(self, **args): - return self.reentrant and super().shouldtest(**args) - - def test(self, exec=[], persist=False, gdb=False, failure=None, **args): - for cycles in it.count(1): - # clear disk first? - if cycles == 1 and persist != 'noerase': - persist = 'erase' - else: - persist = 'noerase' - - # exact cycle we should drop into debugger? - if gdb and failure and failure.cycleno == cycles: - return super().test(gdb=gdb, persist=persist, cycles=cycles, - failure=failure, **args) - - # run tests, but kill the program after prog/erase has - # been hit n cycles. We exit with a special return code if the - # program has not finished, since this isn't a test failure. - try: - return super().test(persist=persist, cycles=cycles, **args) - except TestFailure as nfailure: - if nfailure.returncode == 33: - continue - else: - nfailure.cycleno = cycles - raise - -class TestSuite: - def __init__(self, path, classes=[TestCase], defines={}, - filter=None, **args): - self.name = os.path.basename(path) - if self.name.endswith('.toml'): - self.name = self.name[:-len('.toml')] - self.path = path - self.classes = classes - self.defines = defines.copy() - self.filter = filter - - with open(path) as f: - # load tests - config = toml.load(f) - - # find line numbers - f.seek(0) - linenos = [] - code_linenos = [] - for i, line in enumerate(f): - if re.match(r'\[\[\s*case\s*\]\]', line): - linenos.append(i+1) - if re.match(r'code\s*=\s*(\'\'\'|""")', line): - code_linenos.append(i+2) - - code_linenos.reverse() - - # grab global config - for k, v in config.get('define', {}).items(): - if k not in self.defines: - self.defines[k] = v - self.code = config.get('code', None) - if self.code is not None: - self.code_lineno = code_linenos.pop() - - # create initial test cases - self.cases = [] - for i, (case, lineno) in enumerate(zip(config['case'], linenos)): - # code lineno? - if 'code' in case: - case['code_lineno'] = code_linenos.pop() - # give our case's config a copy of our "global" config - for k, v in config.items(): - if k not in case: - case[k] = v - # initialize test case - self.cases.append(TestCase(case, filter=filter, - suite=self, caseno=i+1, lineno=lineno, **args)) - - def __str__(self): - return self.name - - def __lt__(self, other): - return self.name < other.name - - def permute(self, **args): - for case in self.cases: - # lets find all parameterized definitions, in one of [args.D, - # suite.defines, case.defines, DEFINES]. Note that each of these - # can be either a dict of defines, or a list of dicts, expressing - # an initial set of permutations. - pending = [{}] - for inits in [self.defines, case.defines, DEFINES]: - if not isinstance(inits, list): - inits = [inits] - - npending = [] - for init, pinit in it.product(inits, pending): - ninit = pinit.copy() - for k, v in init.items(): - if k not in ninit: - try: - ninit[k] = eval(v) - except: - ninit[k] = v - npending.append(ninit) - - pending = npending - - # expand permutations - pending = list(reversed(pending)) - expanded = [] - while pending: - perm = pending.pop() - for k, v in sorted(perm.items()): - if not isinstance(v, str) and isinstance(v, abc.Iterable): - for nv in reversed(v): - nperm = perm.copy() - nperm[k] = nv - pending.append(nperm) - break - else: - expanded.append(perm) - - # generate permutations - case.perms = [] - for i, (class_, defines) in enumerate( - it.product(self.classes, expanded)): - case.perms.append(case.permute( - class_, defines, permno=i+1, **args)) - - # also track non-unique defines - case.defines = {} - for k, v in case.perms[0].defines.items(): - if all(perm.defines[k] == v for perm in case.perms): - case.defines[k] = v - - # track all perms and non-unique defines - self.perms = [] - for case in self.cases: - self.perms.extend(case.perms) - - self.defines = {} - for k, v in self.perms[0].defines.items(): - if all(perm.defines.get(k, None) == v for perm in self.perms): - self.defines[k] = v - - return self.perms - - def build(self, **args): - # build test files - tf = open(self.path + '.test.c.t', 'w') - tf.write(GLOBALS) - if self.code is not None: - tf.write('#line %d "%s"\n' % (self.code_lineno, self.path)) - tf.write(self.code) - - tfs = {None: tf} - for case in self.cases: - if case.in_ not in tfs: - tfs[case.in_] = open(self.path+'.'+ - case.in_.replace('/', '.')+'.t', 'w') - tfs[case.in_].write('#line 1 "%s"\n' % case.in_) - with open(case.in_) as f: - for line in f: - tfs[case.in_].write(line) - tfs[case.in_].write('\n') - tfs[case.in_].write(GLOBALS) - - tfs[case.in_].write('\n') - case.build(tfs[case.in_], **args) - - tf.write('\n') - tf.write('const char *lfs_testbd_path;\n') - tf.write('uint32_t lfs_testbd_cycles;\n') - tf.write('int main(int argc, char **argv) {\n') - tf.write(4*' '+'int case_ = (argc > 1) ? atoi(argv[1]) : 0;\n') - tf.write(4*' '+'int perm = (argc > 2) ? atoi(argv[2]) : 0;\n') - tf.write(4*' '+'lfs_testbd_path = (argc > 3) ? argv[3] : NULL;\n') - tf.write(4*' '+'lfs_testbd_cycles = (argc > 4) ? atoi(argv[4]) : 0;\n') - for perm in self.perms: - # test declaration - tf.write(4*' '+'extern void test_case%d(%s);\n' % ( - perm.caseno, ', '.join( - 'intmax_t %s' % k for k in sorted(perm.defines) - if k not in perm.case.defines))) - # test call - tf.write(4*' '+ - 'if (argc < 3 || (case_ == %d && perm == %d)) {' - ' test_case%d(%s); ' - '}\n' % (perm.caseno, perm.permno, perm.caseno, ', '.join( - str(v) for k, v in sorted(perm.defines.items()) - if k not in perm.case.defines))) - tf.write('}\n') - - for tf in tfs.values(): - tf.close() - - # write makefiles - with open(self.path + '.mk', 'w') as mk: - mk.write(RULES.replace(4*' ', '\t')) - mk.write('\n') - - # add truely global defines globally - for k, v in sorted(self.defines.items()): - mk.write('%s: override CFLAGS += -D%s=%r\n' % ( - self.path+'.test', k, v)) - - for path in tfs: - if path is None: - mk.write('%s: %s | %s\n' % ( - self.path+'.test.c', - self.path, - self.path+'.test.c.t')) - else: - mk.write('%s: %s %s | %s\n' % ( - self.path+'.'+path.replace('/', '.'), - self.path, path, - self.path+'.'+path.replace('/', '.')+'.t')) - mk.write('\t./scripts/explode_asserts.py $| -o $@\n') - - self.makefile = self.path + '.mk' - self.target = self.path + '.test' - return self.makefile, self.target - - def test(self, **args): - # run test suite! - if not args.get('verbose', True): - sys.stdout.write(self.name + ' ') - sys.stdout.flush() - for perm in self.perms: - if not perm.shouldtest(**args): - continue - - try: - result = perm.test(**args) - except TestFailure as failure: - perm.result = failure - if not args.get('verbose', True): - sys.stdout.write(FAIL) - sys.stdout.flush() - if not args.get('keep_going', False): - if not args.get('verbose', True): - sys.stdout.write('\n') - raise - else: - perm.result = PASS - if not args.get('verbose', True): - sys.stdout.write(PASS) - sys.stdout.flush() - - if not args.get('verbose', True): - sys.stdout.write('\n') - -def main(**args): - # figure out explicit defines - defines = {} - for define in args['D']: - k, v, *_ = define.split('=', 2) + [''] - defines[k] = v - - # and what class of TestCase to run - classes = [] - if args.get('normal', False): - classes.append(TestCase) - if args.get('reentrant', False): - classes.append(ReentrantTestCase) - if args.get('valgrind', False): - classes.append(ValgrindTestCase) - if not classes: - classes = [TestCase] - - suites = [] - for testpath in args['testpaths']: - # optionally specified test case/perm - testpath, *filter = testpath.split('#') - filter = [int(f) for f in filter] - - # figure out the suite's toml file - if os.path.isdir(testpath): - testpath = testpath + '/test_*.toml' - elif os.path.isfile(testpath): - testpath = testpath - elif testpath.endswith('.toml'): - testpath = TESTDIR + '/' + testpath - else: - testpath = TESTDIR + '/' + testpath + '.toml' - - # find tests - for path in glob.glob(testpath): - suites.append(TestSuite(path, classes, defines, filter, **args)) - - # sort for reproducability - suites = sorted(suites) - - # generate permutations - for suite in suites: - suite.permute(**args) - - # build tests in parallel - print('====== building ======') - makefiles = [] - targets = [] - for suite in suites: - makefile, target = suite.build(**args) - makefiles.append(makefile) - targets.append(target) - - cmd = (['make', '-f', 'Makefile'] + - list(it.chain.from_iterable(['-f', m] for m in makefiles)) + - [target for target in targets]) - mpty, spty = pty.openpty() - if args.get('verbose', False): - print(' '.join(shlex.quote(c) for c in cmd)) - proc = sp.Popen(cmd, stdout=spty, stderr=spty) - os.close(spty) - mpty = os.fdopen(mpty, 'r', 1) - stdout = [] - while True: - try: - line = mpty.readline() - except OSError as e: - if e.errno == errno.EIO: - break - raise - stdout.append(line) - if args.get('verbose', False): - sys.stdout.write(line) - # intercept warnings - m = re.match( - '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' - .format('(?:\033\[[\d;]*.| )*', 'warning'), - line) - if m and not args.get('verbose', False): - try: - with open(m.group(1)) as f: - lineno = int(m.group(2)) - line = next(it.islice(f, lineno-1, None)).strip('\n') - sys.stdout.write( - "\033[01m{path}:{lineno}:\033[01;35mwarning:\033[m " - "{message}\n{line}\n\n".format( - path=m.group(1), line=line, lineno=lineno, - message=m.group(3))) - except: - pass - proc.wait() - - if proc.returncode != 0: - if not args.get('verbose', False): - for line in stdout: - sys.stdout.write(line) - sys.exit(-3) - - print('built %d test suites, %d test cases, %d permutations' % ( - len(suites), - sum(len(suite.cases) for suite in suites), - sum(len(suite.perms) for suite in suites))) - - filtered = 0 - for suite in suites: - for perm in suite.perms: - filtered += perm.shouldtest(**args) - if filtered != sum(len(suite.perms) for suite in suites): - print('filtered down to %d permutations' % filtered) - - print('====== testing ======') - try: - for suite in suites: - suite.test(**args) - except TestFailure: - pass - - print('====== results ======') - passed = 0 - failed = 0 - for suite in suites: - for perm in suite.perms: - if not hasattr(perm, 'result'): - continue - - if perm.result == PASS: - passed += 1 - else: - sys.stdout.write( - "\033[01m{path}:{lineno}:\033[01;31mfailure:\033[m " - "{perm} failed with {returncode}\n".format( - perm=perm, path=perm.suite.path, lineno=perm.lineno, - returncode=perm.result.returncode or 0)) - if perm.result.stdout: - for line in (perm.result.stdout - if not perm.result.assert_ - else perm.result.stdout[:-1]): - sys.stdout.write(line) - if perm.result.assert_: - sys.stdout.write( - "\033[01m{path}:{lineno}:\033[01;31massert:\033[m " - "{message}\n{line}\n".format( - **perm.result.assert_)) - else: - for line in perm.result.stdout: - sys.stdout.write(line) - sys.stdout.write('\n') - failed += 1 - - if args.get('gdb', False): - failure = None - for suite in suites: - for perm in suite.perms: - if getattr(perm, 'result', PASS) != PASS: - failure = perm.result - if failure is not None: - print('======= gdb ======') - # drop into gdb - failure.case.test(failure=failure, **args) - sys.exit(0) - - print('tests passed: %d' % passed) - print('tests failed: %d' % failed) - return 1 if failed > 0 else 0 - -if __name__ == "__main__": - import argparse - parser = argparse.ArgumentParser( - description="Run parameterized tests in various configurations.") - parser.add_argument('testpaths', nargs='*', default=[TESTDIR], - help="Description of test(s) to run. By default, this is all tests \ - found in the \"{0}\" directory. Here, you can specify a different \ - directory of tests, a specific file, a suite by name, and even a \ - specific test case by adding brackets. For example \ - \"test_dirs[0]\" or \"{0}/test_dirs.toml[0]\".".format(TESTDIR)) - parser.add_argument('-D', action='append', default=[], - help="Overriding parameter definitions.") - parser.add_argument('-v', '--verbose', action='store_true', - help="Output everything that is happening.") - parser.add_argument('-k', '--keep-going', action='store_true', - help="Run all tests instead of stopping on first error. Useful for CI.") - parser.add_argument('-p', '--persist', choices=['erase', 'noerase'], - nargs='?', const='erase', - help="Store disk image in a file.") - parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], - nargs='?', const='assert', - help="Drop into gdb on test failure.") - parser.add_argument('--no-internal', action='store_true', - help="Don't run tests that require internal knowledge.") - parser.add_argument('-n', '--normal', action='store_true', - help="Run tests normally.") - parser.add_argument('-r', '--reentrant', action='store_true', - help="Run reentrant tests with simulated power-loss.") - parser.add_argument('-V', '--valgrind', action='store_true', - help="Run non-leaky tests under valgrind to check for memory leaks.") - parser.add_argument('-e', '--exec', default=[], type=lambda e: e.split(' '), - help="Run tests with another executable prefixed on the command line.") - sys.exit(main(**vars(parser.parse_args()))) diff --git a/tests/test_alloc.sh b/tests/test_alloc.sh deleted file mode 100755 index 06698508..00000000 --- a/tests/test_alloc.sh +++ /dev/null @@ -1,494 +0,0 @@ -#!/bin/bash -set -euE -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Allocator tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -SIZE=15000 - -lfs_mkdir() { -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "$1") => 0; - lfs_unmount(&lfs) => 0; -TEST -} - -lfs_remove() { -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_remove(&lfs, "$1/eggs") => 0; - lfs_remove(&lfs, "$1/bacon") => 0; - lfs_remove(&lfs, "$1/pancakes") => 0; - lfs_remove(&lfs, "$1") => 0; - lfs_unmount(&lfs) => 0; -TEST -} - -lfs_alloc_singleproc() { -scripts/test.py << TEST - const char *names[] = {"bacon", "eggs", "pancakes"}; - lfs_file_t files[sizeof(names)/sizeof(names[0])]; - lfs_mount(&lfs, &cfg) => 0; - for (unsigned n = 0; n < sizeof(names)/sizeof(names[0]); n++) { - sprintf(path, "$1/%s", names[n]); - lfs_file_open(&lfs, &files[n], path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; - } - for (unsigned n = 0; n < sizeof(names)/sizeof(names[0]); n++) { - lfs_size_t size = strlen(names[n]); - for (int i = 0; i < $SIZE; i++) { - lfs_file_write(&lfs, &files[n], names[n], size) => size; - } - } - for (unsigned n = 0; n < sizeof(names)/sizeof(names[0]); n++) { - lfs_file_close(&lfs, &files[n]) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -} - -lfs_alloc_multiproc() { -for name in bacon eggs pancakes -do -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "$1/$name", - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; - lfs_size_t size = strlen("$name"); - memcpy(buffer, "$name", size); - for (int i = 0; i < $SIZE; i++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -done -} - -lfs_verify() { -for name in bacon eggs pancakes -do -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "$1/$name", LFS_O_RDONLY) => 0; - lfs_size_t size = strlen("$name"); - for (int i = 0; i < $SIZE; i++) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "$name", size) => 0; - } - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -done -} - -echo "--- Single-process allocation test ---" -lfs_mkdir singleproc -lfs_alloc_singleproc singleproc -lfs_verify singleproc - -echo "--- Multi-process allocation test ---" -lfs_mkdir multiproc -lfs_alloc_multiproc multiproc -lfs_verify multiproc -lfs_verify singleproc - -echo "--- Single-process reuse test ---" -lfs_remove singleproc -lfs_mkdir singleprocreuse -lfs_alloc_singleproc singleprocreuse -lfs_verify singleprocreuse -lfs_verify multiproc - -echo "--- Multi-process reuse test ---" -lfs_remove multiproc -lfs_mkdir multiprocreuse -lfs_alloc_singleproc multiprocreuse -lfs_verify multiprocreuse -lfs_verify singleprocreuse - -echo "--- Cleanup ---" -lfs_remove multiprocreuse -lfs_remove singleprocreuse - -echo "--- Exhaustion test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - lfs_size_t size = strlen("exhaustion"); - memcpy(buffer, "exhaustion", size); - lfs_file_write(&lfs, &file, buffer, size) => size; - lfs_file_sync(&lfs, &file) => 0; - - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - lfs_ssize_t res; - while (true) { - res = lfs_file_write(&lfs, &file, buffer, size); - if (res < 0) { - break; - } - - res => size; - } - res => LFS_ERR_NOSPC; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY); - lfs_size_t size = strlen("exhaustion"); - lfs_file_size(&lfs, &file) => size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "exhaustion", size) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Exhaustion wraparound test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_remove(&lfs, "exhaustion") => 0; - - lfs_file_open(&lfs, &file, "padding", LFS_O_WRONLY | LFS_O_CREAT); - lfs_size_t size = strlen("buffering"); - memcpy(buffer, "buffering", size); - for (int i = 0; i < $SIZE; i++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - lfs_remove(&lfs, "padding") => 0; - - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - size = strlen("exhaustion"); - memcpy(buffer, "exhaustion", size); - lfs_file_write(&lfs, &file, buffer, size) => size; - lfs_file_sync(&lfs, &file) => 0; - - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - lfs_ssize_t res; - while (true) { - res = lfs_file_write(&lfs, &file, buffer, size); - if (res < 0) { - break; - } - - res => size; - } - res => LFS_ERR_NOSPC; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_RDONLY); - lfs_size_t size = strlen("exhaustion"); - lfs_file_size(&lfs, &file) => size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "exhaustion", size) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_remove(&lfs, "exhaustion") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Dir exhaustion test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - // find out max file size - lfs_mkdir(&lfs, "exhaustiondir") => 0; - lfs_size_t size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - int count = 0; - int err; - while (true) { - err = lfs_file_write(&lfs, &file, buffer, size); - if (err < 0) { - break; - } - - count += 1; - } - err => LFS_ERR_NOSPC; - lfs_file_close(&lfs, &file) => 0; - - lfs_remove(&lfs, "exhaustion") => 0; - lfs_remove(&lfs, "exhaustiondir") => 0; - - // see if dir fits with max file size - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - for (int i = 0; i < count; i++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - lfs_mkdir(&lfs, "exhaustiondir") => 0; - lfs_remove(&lfs, "exhaustiondir") => 0; - lfs_remove(&lfs, "exhaustion") => 0; - - // see if dir fits with > max file size - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - for (int i = 0; i < count+1; i++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - lfs_mkdir(&lfs, "exhaustiondir") => LFS_ERR_NOSPC; - - lfs_remove(&lfs, "exhaustion") => 0; - lfs_unmount(&lfs) => 0; -TEST - -## Below, these tests depend _very_ heavily on the geometry of the -## block device being tested, they should be removed and replaced -## by generalized tests. For now we'll just skip if the geometry -## is customized. - -if [[ ! $MAKEFLAGS =~ "LFS_BLOCK_CYCLES" ]] -then - -echo "--- Chained dir exhaustion test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - // find out max file size - lfs_mkdir(&lfs, "exhaustiondir") => 0; - for (int i = 0; i < 10; i++) { - sprintf(path, "dirwithanexhaustivelylongnameforpadding%d", i); - lfs_mkdir(&lfs, path) => 0; - } - lfs_size_t size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - int count = 0; - int err; - while (true) { - err = lfs_file_write(&lfs, &file, buffer, size); - if (err < 0) { - break; - } - - count += 1; - } - err => LFS_ERR_NOSPC; - lfs_file_close(&lfs, &file) => 0; - - lfs_remove(&lfs, "exhaustion") => 0; - lfs_remove(&lfs, "exhaustiondir") => 0; - for (int i = 0; i < 10; i++) { - sprintf(path, "dirwithanexhaustivelylongnameforpadding%d", i); - lfs_remove(&lfs, path) => 0; - } - - // see that chained dir fails - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - for (int i = 0; i < count+1; i++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_sync(&lfs, &file) => 0; - - for (int i = 0; i < 10; i++) { - sprintf(path, "dirwithanexhaustivelylongnameforpadding%d", i); - lfs_mkdir(&lfs, path) => 0; - } - - lfs_mkdir(&lfs, "exhaustiondir") => LFS_ERR_NOSPC; - - // shorten file to try a second chained dir - while (true) { - err = lfs_mkdir(&lfs, "exhaustiondir"); - if (err != LFS_ERR_NOSPC) { - break; - } - - lfs_ssize_t filesize = lfs_file_size(&lfs, &file); - filesize > 0 => true; - - lfs_file_truncate(&lfs, &file, filesize - size) => 0; - lfs_file_sync(&lfs, &file) => 0; - } - err => 0; - - lfs_mkdir(&lfs, "exhaustiondir2") => LFS_ERR_NOSPC; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Split dir test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - // create one block hole for half a directory - lfs_file_open(&lfs, &file, "bump", LFS_O_WRONLY | LFS_O_CREAT) => 0; - for (lfs_size_t i = 0; i < cfg.block_size; i += 2) { - memcpy(&buffer[i], "hi", 2); - } - lfs_file_write(&lfs, &file, buffer, cfg.block_size) => cfg.block_size; - lfs_file_close(&lfs, &file) => 0; - - lfs_file_open(&lfs, &file, "exhaustion", LFS_O_WRONLY | LFS_O_CREAT); - lfs_size_t size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < (cfg.block_count-4)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - // remount to force reset of lookahead - lfs_unmount(&lfs) => 0; - lfs_mount(&lfs, &cfg) => 0; - - // open hole - lfs_remove(&lfs, "bump") => 0; - - lfs_mkdir(&lfs, "splitdir") => 0; - lfs_file_open(&lfs, &file, "splitdir/bump", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - for (lfs_size_t i = 0; i < cfg.block_size; i += 2) { - memcpy(&buffer[i], "hi", 2); - } - lfs_file_write(&lfs, &file, buffer, 2*cfg.block_size) => LFS_ERR_NOSPC; - lfs_file_close(&lfs, &file) => 0; - - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Outdated lookahead test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - - // fill completely with two files - lfs_file_open(&lfs, &file, "exhaustion1", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_size_t size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2)/2)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - lfs_file_open(&lfs, &file, "exhaustion2", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2+1)/2)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - // remount to force reset of lookahead - lfs_unmount(&lfs) => 0; - lfs_mount(&lfs, &cfg) => 0; - - // rewrite one file - lfs_file_open(&lfs, &file, "exhaustion1", - LFS_O_WRONLY | LFS_O_TRUNC) => 0; - lfs_file_sync(&lfs, &file) => 0; - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2)/2)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - // rewrite second file, this requires lookahead does not - // use old population - lfs_file_open(&lfs, &file, "exhaustion2", - LFS_O_WRONLY | LFS_O_TRUNC) => 0; - lfs_file_sync(&lfs, &file) => 0; - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2+1)/2)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; -TEST - -echo "--- Outdated lookahead and split dir test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - - // fill completely with two files - lfs_file_open(&lfs, &file, "exhaustion1", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_size_t size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2)/2)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - lfs_file_open(&lfs, &file, "exhaustion2", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2+1)/2)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - // remount to force reset of lookahead - lfs_unmount(&lfs) => 0; - lfs_mount(&lfs, &cfg) => 0; - - // rewrite one file with a hole of one block - lfs_file_open(&lfs, &file, "exhaustion1", - LFS_O_WRONLY | LFS_O_TRUNC) => 0; - lfs_file_sync(&lfs, &file) => 0; - size = strlen("blahblahblahblah"); - memcpy(buffer, "blahblahblahblah", size); - for (lfs_size_t i = 0; - i < ((cfg.block_count-2)/2 - 1)*(cfg.block_size-8); - i += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_close(&lfs, &file) => 0; - - // try to allocate a directory, should fail! - lfs_mkdir(&lfs, "split") => LFS_ERR_NOSPC; - - // file should not fail - lfs_file_open(&lfs, &file, "notasplit", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_write(&lfs, &file, "hi", 2) => 2; - lfs_file_close(&lfs, &file) => 0; - - lfs_unmount(&lfs) => 0; -TEST - -fi - -scripts/results.py diff --git a/tests_/test_alloc.toml b/tests/test_alloc.toml similarity index 100% rename from tests_/test_alloc.toml rename to tests/test_alloc.toml diff --git a/tests/test_attrs.sh b/tests/test_attrs.sh deleted file mode 100755 index 612cae1e..00000000 --- a/tests/test_attrs.sh +++ /dev/null @@ -1,294 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Attr tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "hello") => 0; - lfs_file_open(&lfs, &file, "hello/hello", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_write(&lfs, &file, "hello", strlen("hello")) - => strlen("hello"); - lfs_file_close(&lfs, &file); - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Set/get attribute ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - lfs_setattr(&lfs, "hello", 'A', "aaaa", 4) => 0; - lfs_setattr(&lfs, "hello", 'B', "bbbbbb", 6) => 0; - lfs_setattr(&lfs, "hello", 'C', "ccccc", 5) => 0; - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 6; - lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "bbbbbb", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "hello", 'B', "", 0) => 0; - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 0; - lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_removeattr(&lfs, "hello", 'B') => 0; - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => LFS_ERR_NOATTR; - lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "hello", 'B', "dddddd", 6) => 0; - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 6; - lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "dddddd", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "hello", 'B', "eee", 3) => 0; - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 3; - lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "eee\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "hello", 'A', buffer, LFS_ATTR_MAX+1) => LFS_ERR_NOSPC; - lfs_setattr(&lfs, "hello", 'B', "fffffffff", 9) => 0; - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 6) => 9; - lfs_getattr(&lfs, "hello", 'C', buffer+10, 5) => 5; - - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - lfs_getattr(&lfs, "hello", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "hello", 'B', buffer+4, 9) => 9; - lfs_getattr(&lfs, "hello", 'C', buffer+13, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "fffffffff", 9) => 0; - memcmp(buffer+13, "ccccc", 5) => 0; - - lfs_file_open(&lfs, &file, "hello/hello", LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => strlen("hello"); - memcmp(buffer, "hello", strlen("hello")) => 0; - lfs_file_close(&lfs, &file); - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Set/get root attribute ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - lfs_setattr(&lfs, "/", 'A', "aaaa", 4) => 0; - lfs_setattr(&lfs, "/", 'B', "bbbbbb", 6) => 0; - lfs_setattr(&lfs, "/", 'C', "ccccc", 5) => 0; - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 6; - lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "bbbbbb", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "/", 'B', "", 0) => 0; - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 0; - lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_removeattr(&lfs, "/", 'B') => 0; - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => LFS_ERR_NOATTR; - lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "/", 'B', "dddddd", 6) => 0; - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 6; - lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "dddddd", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "/", 'B', "eee", 3) => 0; - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 3; - lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "eee\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - lfs_setattr(&lfs, "/", 'A', buffer, LFS_ATTR_MAX+1) => LFS_ERR_NOSPC; - lfs_setattr(&lfs, "/", 'B', "fffffffff", 9) => 0; - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 6) => 9; - lfs_getattr(&lfs, "/", 'C', buffer+10, 5) => 5; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - lfs_getattr(&lfs, "/", 'A', buffer, 4) => 4; - lfs_getattr(&lfs, "/", 'B', buffer+4, 9) => 9; - lfs_getattr(&lfs, "/", 'C', buffer+13, 5) => 5; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "fffffffff", 9) => 0; - memcmp(buffer+13, "ccccc", 5) => 0; - - lfs_file_open(&lfs, &file, "hello/hello", LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => strlen("hello"); - memcmp(buffer, "hello", strlen("hello")) => 0; - lfs_file_close(&lfs, &file); - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Set/get file attribute ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - struct lfs_attr attrs1[] = { - {'A', buffer, 4}, - {'B', buffer+4, 6}, - {'C', buffer+10, 5}, - }; - struct lfs_file_config cfg1 = {.attrs=attrs1, .attr_count=3}; - - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; - memcpy(buffer, "aaaa", 4); - memcpy(buffer+4, "bbbbbb", 6); - memcpy(buffer+10, "ccccc", 5); - lfs_file_close(&lfs, &file) => 0; - memset(buffer, 0, 15); - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; - lfs_file_close(&lfs, &file) => 0; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "bbbbbb", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - attrs1[1].size = 0; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; - lfs_file_close(&lfs, &file) => 0; - memset(buffer, 0, 15); - attrs1[1].size = 6; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; - lfs_file_close(&lfs, &file) => 0; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "\0\0\0\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - attrs1[1].size = 6; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; - memcpy(buffer+4, "dddddd", 6); - lfs_file_close(&lfs, &file) => 0; - memset(buffer, 0, 15); - attrs1[1].size = 6; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; - lfs_file_close(&lfs, &file) => 0; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "dddddd", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - attrs1[1].size = 3; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; - memcpy(buffer+4, "eee", 3); - lfs_file_close(&lfs, &file) => 0; - memset(buffer, 0, 15); - attrs1[1].size = 6; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; - lfs_file_close(&lfs, &file) => 0; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "eee\0\0\0", 6) => 0; - memcmp(buffer+10, "ccccc", 5) => 0; - - attrs1[0].size = LFS_ATTR_MAX+1; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) - => LFS_ERR_NOSPC; - - struct lfs_attr attrs2[] = { - {'A', buffer, 4}, - {'B', buffer+4, 9}, - {'C', buffer+13, 5}, - }; - struct lfs_file_config cfg2 = {.attrs=attrs2, .attr_count=3}; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDWR, &cfg2) => 0; - memcpy(buffer+4, "fffffffff", 9); - lfs_file_close(&lfs, &file) => 0; - attrs1[0].size = 4; - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg1) => 0; - lfs_file_close(&lfs, &file) => 0; - - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - struct lfs_attr attrs2[] = { - {'A', buffer, 4}, - {'B', buffer+4, 9}, - {'C', buffer+13, 5}, - }; - struct lfs_file_config cfg2 = {.attrs=attrs2, .attr_count=3}; - - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_RDONLY, &cfg2) => 0; - lfs_file_close(&lfs, &file) => 0; - memcmp(buffer, "aaaa", 4) => 0; - memcmp(buffer+4, "fffffffff", 9) => 0; - memcmp(buffer+13, "ccccc", 5) => 0; - - lfs_file_open(&lfs, &file, "hello/hello", LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, buffer, sizeof(buffer)) => strlen("hello"); - memcmp(buffer, "hello", strlen("hello")) => 0; - lfs_file_close(&lfs, &file); - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Deferred file attributes ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(buffer, 0, sizeof(buffer)); - struct lfs_attr attrs1[] = { - {'B', "gggg", 4}, - {'C', "", 0}, - {'D', "hhhh", 4}, - }; - struct lfs_file_config cfg1 = {.attrs=attrs1, .attr_count=3}; - - lfs_file_opencfg(&lfs, &file, "hello/hello", LFS_O_WRONLY, &cfg1) => 0; - - lfs_getattr(&lfs, "hello/hello", 'B', buffer, 9) => 9; - lfs_getattr(&lfs, "hello/hello", 'C', buffer+9, 9) => 5; - lfs_getattr(&lfs, "hello/hello", 'D', buffer+18, 9) => LFS_ERR_NOATTR; - memcmp(buffer, "fffffffff", 9) => 0; - memcmp(buffer+9, "ccccc\0\0\0\0", 9) => 0; - memcmp(buffer+18, "\0\0\0\0\0\0\0\0\0", 9) => 0; - - lfs_file_sync(&lfs, &file) => 0; - lfs_getattr(&lfs, "hello/hello", 'B', buffer, 9) => 4; - lfs_getattr(&lfs, "hello/hello", 'C', buffer+9, 9) => 0; - lfs_getattr(&lfs, "hello/hello", 'D', buffer+18, 9) => 4; - memcmp(buffer, "gggg\0\0\0\0\0", 9) => 0; - memcmp(buffer+9, "\0\0\0\0\0\0\0\0\0", 9) => 0; - memcmp(buffer+18, "hhhh\0\0\0\0\0", 9) => 0; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_attrs.toml b/tests/test_attrs.toml similarity index 100% rename from tests_/test_attrs.toml rename to tests/test_attrs.toml diff --git a/tests_/test_badblocks.toml b/tests/test_badblocks.toml similarity index 100% rename from tests_/test_badblocks.toml rename to tests/test_badblocks.toml diff --git a/tests/test_corrupt.sh b/tests/test_corrupt.sh deleted file mode 100755 index 30015224..00000000 --- a/tests/test_corrupt.sh +++ /dev/null @@ -1,120 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Corrupt tests ===" - -NAMEMULT=64 -FILEMULT=1 - -lfs_mktree() { -scripts/test.py ${1:-} << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - for (int i = 1; i < 10; i++) { - for (int j = 0; j < $NAMEMULT; j++) { - buffer[j] = '0'+i; - } - buffer[$NAMEMULT] = '\0'; - lfs_mkdir(&lfs, (char*)buffer) => 0; - - buffer[$NAMEMULT] = '/'; - for (int j = 0; j < $NAMEMULT; j++) { - buffer[j+$NAMEMULT+1] = '0'+i; - } - buffer[2*$NAMEMULT+1] = '\0'; - lfs_file_open(&lfs, &file, (char*)buffer, - LFS_O_WRONLY | LFS_O_CREAT) => 0; - - lfs_size_t size = $NAMEMULT; - for (int j = 0; j < i*$FILEMULT; j++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -} - -lfs_chktree() { -scripts/test.py ${1:-} << TEST - lfs_mount(&lfs, &cfg) => 0; - for (int i = 1; i < 10; i++) { - for (int j = 0; j < $NAMEMULT; j++) { - buffer[j] = '0'+i; - } - buffer[$NAMEMULT] = '\0'; - lfs_stat(&lfs, (char*)buffer, &info) => 0; - info.type => LFS_TYPE_DIR; - - buffer[$NAMEMULT] = '/'; - for (int j = 0; j < $NAMEMULT; j++) { - buffer[j+$NAMEMULT+1] = '0'+i; - } - buffer[2*$NAMEMULT+1] = '\0'; - lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; - - lfs_size_t size = $NAMEMULT; - for (int j = 0; j < i*$FILEMULT; j++) { - uint8_t rbuffer[1024]; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(buffer, rbuffer, size) => 0; - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -} - -echo "--- Sanity check ---" -rm -rf blocks -lfs_mktree -lfs_chktree -BLOCKS="$(ls blocks | grep -vw '[01]')" - -echo "--- Block corruption ---" -for b in $BLOCKS -do - rm -rf blocks - mkdir blocks - ln -s /dev/zero blocks/$b - lfs_mktree - lfs_chktree -done - -echo "--- Block persistance ---" -for b in $BLOCKS -do - rm -rf blocks - mkdir blocks - lfs_mktree - chmod a-w blocks/$b || true - lfs_mktree - lfs_chktree -done - -echo "--- Big region corruption ---" -rm -rf blocks -mkdir blocks -for i in {2..512} -do - ln -s /dev/zero blocks/$(printf '%x' $i) -done -lfs_mktree -lfs_chktree - -echo "--- Alternating corruption ---" -rm -rf blocks -mkdir blocks -for i in {2..1024..2} -do - ln -s /dev/zero blocks/$(printf '%x' $i) -done -lfs_mktree -lfs_chktree - -scripts/results.py diff --git a/tests/test_dirs.sh b/tests/test_dirs.sh deleted file mode 100755 index 0125bfd4..00000000 --- a/tests/test_dirs.sh +++ /dev/null @@ -1,489 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Directory tests ===" - -LARGESIZE=128 - -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -echo "--- Root directory ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Directory creation ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "potato") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- File creation ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "burito", LFS_O_CREAT | LFS_O_WRONLY) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Directory iteration ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "burito") => 0; - info.type => LFS_TYPE_REG; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "potato") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Directory failures ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "potato") => LFS_ERR_EXIST; - lfs_dir_open(&lfs, &dir, "tomato") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "burito") => LFS_ERR_NOTDIR; - lfs_file_open(&lfs, &file, "tomato", LFS_O_RDONLY) => LFS_ERR_NOENT; - lfs_file_open(&lfs, &file, "potato", LFS_O_RDONLY) => LFS_ERR_ISDIR; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Nested directories ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "potato/baked") => 0; - lfs_mkdir(&lfs, "potato/sweet") => 0; - lfs_mkdir(&lfs, "potato/fried") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "potato") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "baked") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "fried") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "sweet") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Multi-block directory ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "cactus") => 0; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "cactus/test%03d", i); - lfs_mkdir(&lfs, path) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "cactus") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "test%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.type => LFS_TYPE_DIR; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Directory remove ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_remove(&lfs, "potato") => LFS_ERR_NOTEMPTY; - lfs_remove(&lfs, "potato/sweet") => 0; - lfs_remove(&lfs, "potato/baked") => 0; - lfs_remove(&lfs, "potato/fried") => 0; - - lfs_dir_open(&lfs, &dir, "potato") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_remove(&lfs, "potato") => 0; - - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "burito") => 0; - info.type => LFS_TYPE_REG; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "cactus") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "burito") => 0; - info.type => LFS_TYPE_REG; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "cactus") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Directory rename ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "coldpotato") => 0; - lfs_mkdir(&lfs, "coldpotato/baked") => 0; - lfs_mkdir(&lfs, "coldpotato/sweet") => 0; - lfs_mkdir(&lfs, "coldpotato/fried") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "coldpotato", "hotpotato") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "hotpotato") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "baked") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "fried") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "sweet") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "warmpotato") => 0; - lfs_mkdir(&lfs, "warmpotato/mushy") => 0; - lfs_rename(&lfs, "hotpotato", "warmpotato") => LFS_ERR_NOTEMPTY; - - lfs_remove(&lfs, "warmpotato/mushy") => 0; - lfs_rename(&lfs, "hotpotato", "warmpotato") => 0; - - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "warmpotato") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "baked") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "fried") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "sweet") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "coldpotato") => 0; - lfs_rename(&lfs, "warmpotato/baked", "coldpotato/baked") => 0; - lfs_rename(&lfs, "warmpotato/sweet", "coldpotato/sweet") => 0; - lfs_rename(&lfs, "warmpotato/fried", "coldpotato/fried") => 0; - lfs_remove(&lfs, "coldpotato") => LFS_ERR_NOTEMPTY; - lfs_remove(&lfs, "warmpotato") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "coldpotato") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "baked") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "fried") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "sweet") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Recursive remove ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_remove(&lfs, "coldpotato") => LFS_ERR_NOTEMPTY; - - lfs_dir_open(&lfs, &dir, "coldpotato") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - - while (true) { - int err = lfs_dir_read(&lfs, &dir, &info); - err >= 0 => 1; - if (err == 0) { - break; - } - - strcpy(path, "coldpotato/"); - strcat(path, info.name); - lfs_remove(&lfs, path) => 0; - } - - lfs_remove(&lfs, "coldpotato") => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "burito") => 0; - info.type => LFS_TYPE_REG; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "cactus") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Multi-block rename ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - for (int i = 0; i < $LARGESIZE; i++) { - char oldpath[1024]; - char newpath[1024]; - sprintf(oldpath, "cactus/test%03d", i); - sprintf(newpath, "cactus/tedd%03d", i); - lfs_rename(&lfs, oldpath, newpath) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "cactus") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "tedd%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.type => LFS_TYPE_DIR; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Multi-block remove ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_remove(&lfs, "cactus") => LFS_ERR_NOTEMPTY; - - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "cactus/tedd%03d", i); - lfs_remove(&lfs, path) => 0; - } - - lfs_remove(&lfs, "cactus") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "burito") => 0; - info.type => LFS_TYPE_REG; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Multi-block directory with files ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "prickly-pear") => 0; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "prickly-pear/test%03d", i); - lfs_file_open(&lfs, &file, path, LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_size_t size = 6; - memcpy(buffer, "Hello", size); - lfs_file_write(&lfs, &file, buffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "prickly-pear") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "test%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.type => LFS_TYPE_REG; - info.size => 6; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Multi-block rename with files ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - for (int i = 0; i < $LARGESIZE; i++) { - char oldpath[1024]; - char newpath[1024]; - sprintf(oldpath, "prickly-pear/test%03d", i); - sprintf(newpath, "prickly-pear/tedd%03d", i); - lfs_rename(&lfs, oldpath, newpath) => 0; - } - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "prickly-pear") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "tedd%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.type => LFS_TYPE_REG; - info.size => 6; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Multi-block remove with files ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_remove(&lfs, "prickly-pear") => LFS_ERR_NOTEMPTY; - - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "prickly-pear/tedd%03d", i); - lfs_remove(&lfs, path) => 0; - } - - lfs_remove(&lfs, "prickly-pear") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "burito") => 0; - info.type => LFS_TYPE_REG; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_dirs.toml b/tests/test_dirs.toml similarity index 100% rename from tests_/test_dirs.toml rename to tests/test_dirs.toml diff --git a/tests/test_entries.sh b/tests/test_entries.sh deleted file mode 100755 index 5075faf5..00000000 --- a/tests/test_entries.sh +++ /dev/null @@ -1,251 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Entry tests ===" - -# Note: These tests are intended for 512 byte inline size at different -# inline sizes they should still pass, but won't be testing anything - -rm -rf blocks -function read_file { -cat << TEST - - size = $2; - lfs_file_open(&lfs, &file, "$1", LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(rbuffer, wbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; -TEST -} - -function write_file { -cat << TEST - - size = $2; - lfs_file_open(&lfs, &file, "$1", - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; - memset(wbuffer, 'c', size); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; -TEST -} - -echo "--- Entry grow test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - lfs_size_t size; - - lfs_mount(&lfs, &cfg) => 0; - $(write_file "hi0" 20) - $(write_file "hi1" 20) - $(write_file "hi2" 20) - $(write_file "hi3" 20) - - $(read_file "hi1" 20) - $(write_file "hi1" 200) - - $(read_file "hi0" 20) - $(read_file "hi1" 200) - $(read_file "hi2" 20) - $(read_file "hi3" 20) - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Entry shrink test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - lfs_size_t size; - - lfs_mount(&lfs, &cfg) => 0; - $(write_file "hi0" 20) - $(write_file "hi1" 200) - $(write_file "hi2" 20) - $(write_file "hi3" 20) - - $(read_file "hi1" 200) - $(write_file "hi1" 20) - - $(read_file "hi0" 20) - $(read_file "hi1" 20) - $(read_file "hi2" 20) - $(read_file "hi3" 20) - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Entry spill test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - lfs_size_t size; - - lfs_mount(&lfs, &cfg) => 0; - $(write_file "hi0" 200) - $(write_file "hi1" 200) - $(write_file "hi2" 200) - $(write_file "hi3" 200) - - $(read_file "hi0" 200) - $(read_file "hi1" 200) - $(read_file "hi2" 200) - $(read_file "hi3" 200) - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Entry push spill test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - lfs_size_t size; - - lfs_mount(&lfs, &cfg) => 0; - $(write_file "hi0" 200) - $(write_file "hi1" 20) - $(write_file "hi2" 200) - $(write_file "hi3" 200) - - $(read_file "hi1" 20) - $(write_file "hi1" 200) - - $(read_file "hi0" 200) - $(read_file "hi1" 200) - $(read_file "hi2" 200) - $(read_file "hi3" 200) - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Entry push spill two test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - lfs_size_t size; - - lfs_mount(&lfs, &cfg) => 0; - $(write_file "hi0" 200) - $(write_file "hi1" 20) - $(write_file "hi2" 200) - $(write_file "hi3" 200) - $(write_file "hi4" 200) - - $(read_file "hi1" 20) - $(write_file "hi1" 200) - - $(read_file "hi0" 200) - $(read_file "hi1" 200) - $(read_file "hi2" 200) - $(read_file "hi3" 200) - $(read_file "hi4" 200) - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Entry drop test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - lfs_size_t size; - - lfs_mount(&lfs, &cfg) => 0; - $(write_file "hi0" 200) - $(write_file "hi1" 200) - $(write_file "hi2" 200) - $(write_file "hi3" 200) - - lfs_remove(&lfs, "hi1") => 0; - lfs_stat(&lfs, "hi1", &info) => LFS_ERR_NOENT; - $(read_file "hi0" 200) - $(read_file "hi2" 200) - $(read_file "hi3" 200) - - lfs_remove(&lfs, "hi2") => 0; - lfs_stat(&lfs, "hi2", &info) => LFS_ERR_NOENT; - $(read_file "hi0" 200) - $(read_file "hi3" 200) - - lfs_remove(&lfs, "hi3") => 0; - lfs_stat(&lfs, "hi3", &info) => LFS_ERR_NOENT; - $(read_file "hi0" 200) - - lfs_remove(&lfs, "hi0") => 0; - lfs_stat(&lfs, "hi0", &info) => LFS_ERR_NOENT; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Create too big ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - memset(path, 'm', 200); - path[200] = '\0'; - - lfs_size_t size = 400; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; - uint8_t wbuffer[1024]; - memset(wbuffer, 'c', size); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - - size = 400; - lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; - uint8_t rbuffer[1024]; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(rbuffer, wbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Resize too big ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - memset(path, 'm', 200); - path[200] = '\0'; - - lfs_size_t size = 40; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; - uint8_t wbuffer[1024]; - memset(wbuffer, 'c', size); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - - size = 40; - lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; - uint8_t rbuffer[1024]; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(rbuffer, wbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - - size = 400; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; - memset(wbuffer, 'c', size); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - - size = 400; - lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(rbuffer, wbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_entries.toml b/tests/test_entries.toml similarity index 100% rename from tests_/test_entries.toml rename to tests/test_entries.toml diff --git a/tests_/test_exhaustion.toml b/tests/test_exhaustion.toml similarity index 100% rename from tests_/test_exhaustion.toml rename to tests/test_exhaustion.toml diff --git a/tests/test_files.sh b/tests/test_files.sh deleted file mode 100755 index f6535f6a..00000000 --- a/tests/test_files.sh +++ /dev/null @@ -1,221 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== File tests ===" - -SMALLSIZE=32 -MEDIUMSIZE=8192 -LARGESIZE=262144 - -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -echo "--- Simple file test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello", LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_size_t size = strlen("Hello World!\n"); - uint8_t wbuffer[1024]; - memcpy(wbuffer, "Hello World!\n", size); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - - lfs_file_open(&lfs, &file, "hello", LFS_O_RDONLY) => 0; - size = strlen("Hello World!\n"); - uint8_t rbuffer[1024]; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(rbuffer, wbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -w_test() { -scripts/test.py ${4:-} << TEST - lfs_size_t size = $1; - lfs_size_t chunk = 31; - srand(0); - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "$2", - ${3:-LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC}) => 0; - for (lfs_size_t i = 0; i < size; i += chunk) { - chunk = (chunk < size - i) ? chunk : size - i; - for (lfs_size_t b = 0; b < chunk; b++) { - buffer[b] = rand() & 0xff; - } - lfs_file_write(&lfs, &file, buffer, chunk) => chunk; - } - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -} - -r_test() { -scripts/test.py << TEST - lfs_size_t size = $1; - lfs_size_t chunk = 29; - srand(0); - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "$2", &info) => 0; - info.type => LFS_TYPE_REG; - info.size => size; - lfs_file_open(&lfs, &file, "$2", ${3:-LFS_O_RDONLY}) => 0; - for (lfs_size_t i = 0; i < size; i += chunk) { - chunk = (chunk < size - i) ? chunk : size - i; - lfs_file_read(&lfs, &file, buffer, chunk) => chunk; - for (lfs_size_t b = 0; b < chunk && i+b < size; b++) { - buffer[b] => rand() & 0xff; - } - } - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -} - -echo "--- Small file test ---" -w_test $SMALLSIZE smallavacado -r_test $SMALLSIZE smallavacado - -echo "--- Medium file test ---" -w_test $MEDIUMSIZE mediumavacado -r_test $MEDIUMSIZE mediumavacado - -echo "--- Large file test ---" -w_test $LARGESIZE largeavacado -r_test $LARGESIZE largeavacado - -echo "--- Zero file test ---" -w_test 0 noavacado -r_test 0 noavacado - -echo "--- Truncate small test ---" -w_test $SMALLSIZE mediumavacado -r_test $SMALLSIZE mediumavacado -w_test $MEDIUMSIZE mediumavacado -r_test $MEDIUMSIZE mediumavacado - -echo "--- Truncate zero test ---" -w_test $SMALLSIZE noavacado -r_test $SMALLSIZE noavacado -w_test 0 noavacado -r_test 0 noavacado - -echo "--- Non-overlap check ---" -r_test $SMALLSIZE smallavacado -r_test $MEDIUMSIZE mediumavacado -r_test $LARGESIZE largeavacado -r_test 0 noavacado - -echo "--- Dir check ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - info.type => LFS_TYPE_REG; - info.size => strlen("Hello World!\n"); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "largeavacado") => 0; - info.type => LFS_TYPE_REG; - info.size => $LARGESIZE; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "mediumavacado") => 0; - info.type => LFS_TYPE_REG; - info.size => $MEDIUMSIZE; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "noavacado") => 0; - info.type => LFS_TYPE_REG; - info.size => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "smallavacado") => 0; - info.type => LFS_TYPE_REG; - info.size => $SMALLSIZE; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Many files test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST -scripts/test.py << TEST - // Create 300 files of 7 bytes - lfs_mount(&lfs, &cfg) => 0; - for (unsigned i = 0; i < 300; i++) { - sprintf(path, "file_%03d", i); - lfs_file_open(&lfs, &file, path, - LFS_O_RDWR | LFS_O_CREAT | LFS_O_EXCL) => 0; - lfs_size_t size = 7; - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - snprintf((char*)wbuffer, size, "Hi %03d", i); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_rewind(&lfs, &file) => 0; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(wbuffer, rbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Many files with flush test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST -scripts/test.py << TEST - // Create 300 files of 7 bytes - lfs_mount(&lfs, &cfg) => 0; - for (unsigned i = 0; i < 300; i++) { - sprintf(path, "file_%03d", i); - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; - lfs_size_t size = 7; - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - snprintf((char*)wbuffer, size, "Hi %03d", i); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - - lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(wbuffer, rbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Many files with power cycle test ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST -scripts/test.py << TEST - // Create 300 files of 7 bytes - lfs_mount(&lfs, &cfg) => 0; - for (unsigned i = 0; i < 300; i++) { - sprintf(path, "file_%03d", i); - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; - lfs_size_t size = 7; - uint8_t wbuffer[1024]; - uint8_t rbuffer[1024]; - snprintf((char*)wbuffer, size, "Hi %03d", i); - lfs_file_write(&lfs, &file, wbuffer, size) => size; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; - - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(wbuffer, rbuffer, size) => 0; - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_files.toml b/tests/test_files.toml similarity index 100% rename from tests_/test_files.toml rename to tests/test_files.toml diff --git a/tests/test_format.sh b/tests/test_format.sh deleted file mode 100755 index f0972bda..00000000 --- a/tests/test_format.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Formatting tests ===" -rm -rf blocks - -echo "--- Basic formatting ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -echo "--- Basic mounting ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Invalid superblocks ---" -ln -f -s /dev/zero blocks/0 -ln -f -s /dev/zero blocks/1 -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => LFS_ERR_NOSPC; -TEST -rm blocks/0 blocks/1 - -echo "--- Invalid mount ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; -TEST - -echo "--- Expanding superblock ---" -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - lfs_mount(&lfs, &cfg) => 0; - for (int i = 0; i < 100; i++) { - lfs_mkdir(&lfs, "dummy") => 0; - lfs_remove(&lfs, "dummy") => 0; - } - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "dummy") => 0; - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_format.toml b/tests/test_format.toml similarity index 100% rename from tests_/test_format.toml rename to tests/test_format.toml diff --git a/tests/test_interspersed.sh b/tests/test_interspersed.sh deleted file mode 100755 index 84c5dd87..00000000 --- a/tests/test_interspersed.sh +++ /dev/null @@ -1,190 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Interspersed tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -echo "--- Interspersed file test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_t files[4]; - lfs_file_open(&lfs, &files[0], "a", LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_open(&lfs, &files[1], "b", LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_open(&lfs, &files[2], "c", LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_open(&lfs, &files[3], "d", LFS_O_WRONLY | LFS_O_CREAT) => 0; - - for (int i = 0; i < 10; i++) { - lfs_file_write(&lfs, &files[0], (const void*)"a", 1) => 1; - lfs_file_write(&lfs, &files[1], (const void*)"b", 1) => 1; - lfs_file_write(&lfs, &files[2], (const void*)"c", 1) => 1; - lfs_file_write(&lfs, &files[3], (const void*)"d", 1) => 1; - } - - lfs_file_close(&lfs, &files[0]); - lfs_file_close(&lfs, &files[1]); - lfs_file_close(&lfs, &files[2]); - lfs_file_close(&lfs, &files[3]); - - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "a") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "b") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "c") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "d") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_file_open(&lfs, &files[0], "a", LFS_O_RDONLY) => 0; - lfs_file_open(&lfs, &files[1], "b", LFS_O_RDONLY) => 0; - lfs_file_open(&lfs, &files[2], "c", LFS_O_RDONLY) => 0; - lfs_file_open(&lfs, &files[3], "d", LFS_O_RDONLY) => 0; - - for (int i = 0; i < 10; i++) { - lfs_file_read(&lfs, &files[0], buffer, 1) => 1; - buffer[0] => 'a'; - lfs_file_read(&lfs, &files[1], buffer, 1) => 1; - buffer[0] => 'b'; - lfs_file_read(&lfs, &files[2], buffer, 1) => 1; - buffer[0] => 'c'; - lfs_file_read(&lfs, &files[3], buffer, 1) => 1; - buffer[0] => 'd'; - } - - lfs_file_close(&lfs, &files[0]); - lfs_file_close(&lfs, &files[1]); - lfs_file_close(&lfs, &files[2]); - lfs_file_close(&lfs, &files[3]); - - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Interspersed remove file test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_t files[4]; - lfs_file_open(&lfs, &files[0], "e", LFS_O_WRONLY | LFS_O_CREAT) => 0; - - for (int i = 0; i < 5; i++) { - lfs_file_write(&lfs, &files[0], (const void*)"e", 1) => 1; - } - - lfs_remove(&lfs, "a") => 0; - lfs_remove(&lfs, "b") => 0; - lfs_remove(&lfs, "c") => 0; - lfs_remove(&lfs, "d") => 0; - - for (int i = 0; i < 5; i++) { - lfs_file_write(&lfs, &files[0], (const void*)"e", 1) => 1; - } - - lfs_file_close(&lfs, &files[0]); - - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "e") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_file_open(&lfs, &files[0], "e", LFS_O_RDONLY) => 0; - - for (int i = 0; i < 10; i++) { - lfs_file_read(&lfs, &files[0], buffer, 1) => 1; - buffer[0] => 'e'; - } - - lfs_file_close(&lfs, &files[0]); - - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Remove inconveniently test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_t files[4]; - lfs_file_open(&lfs, &files[0], "e", LFS_O_WRONLY | LFS_O_TRUNC) => 0; - lfs_file_open(&lfs, &files[1], "f", LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_open(&lfs, &files[2], "g", LFS_O_WRONLY | LFS_O_CREAT) => 0; - - for (int i = 0; i < 5; i++) { - lfs_file_write(&lfs, &files[0], (const void*)"e", 1) => 1; - lfs_file_write(&lfs, &files[1], (const void*)"f", 1) => 1; - lfs_file_write(&lfs, &files[2], (const void*)"g", 1) => 1; - } - - lfs_remove(&lfs, "f") => 0; - - for (int i = 0; i < 5; i++) { - lfs_file_write(&lfs, &files[0], (const void*)"e", 1) => 1; - lfs_file_write(&lfs, &files[1], (const void*)"f", 1) => 1; - lfs_file_write(&lfs, &files[2], (const void*)"g", 1) => 1; - } - - lfs_file_close(&lfs, &files[0]); - lfs_file_close(&lfs, &files[1]); - lfs_file_close(&lfs, &files[2]); - - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - info.type => LFS_TYPE_DIR; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "e") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "g") => 0; - info.type => LFS_TYPE_REG; - info.size => 10; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_file_open(&lfs, &files[0], "e", LFS_O_RDONLY) => 0; - lfs_file_open(&lfs, &files[1], "g", LFS_O_RDONLY) => 0; - - for (int i = 0; i < 10; i++) { - lfs_file_read(&lfs, &files[0], buffer, 1) => 1; - buffer[0] => 'e'; - lfs_file_read(&lfs, &files[1], buffer, 1) => 1; - buffer[0] => 'g'; - } - - lfs_file_close(&lfs, &files[0]); - lfs_file_close(&lfs, &files[1]); - - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_interspersed.toml b/tests/test_interspersed.toml similarity index 100% rename from tests_/test_interspersed.toml rename to tests/test_interspersed.toml diff --git a/tests/test_move.sh b/tests/test_move.sh deleted file mode 100755 index f52ef22a..00000000 --- a/tests/test_move.sh +++ /dev/null @@ -1,333 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Move tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "a") => 0; - lfs_mkdir(&lfs, "b") => 0; - lfs_mkdir(&lfs, "c") => 0; - lfs_mkdir(&lfs, "d") => 0; - - lfs_mkdir(&lfs, "a/hi") => 0; - lfs_mkdir(&lfs, "a/hi/hola") => 0; - lfs_mkdir(&lfs, "a/hi/bonjour") => 0; - lfs_mkdir(&lfs, "a/hi/ohayo") => 0; - - lfs_file_open(&lfs, &file, "a/hello", LFS_O_CREAT | LFS_O_WRONLY) => 0; - lfs_file_write(&lfs, &file, "hola\n", 5) => 5; - lfs_file_write(&lfs, &file, "bonjour\n", 8) => 8; - lfs_file_write(&lfs, &file, "ohayo\n", 6) => 6; - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move file ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "a/hello", "b/hello") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "a") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "b") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move file corrupt source ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "b/hello", "c/hello") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/corrupt.py -n 1 -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "b") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "c") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move file corrupt source and dest ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "c/hello", "d/hello") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/corrupt.py -n 2 -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "c") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "d") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move file after corrupt ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "c/hello", "d/hello") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "c") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "d") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move dir ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "a/hi", "b/hi") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "a") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "b") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move dir corrupt source ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "b/hi", "c/hi") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/corrupt.py -n 1 -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "b") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "c") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move dir corrupt source and dest ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "c/hi", "d/hi") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/corrupt.py -n 2 -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "c") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "d") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move dir after corrupt ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_rename(&lfs, "c/hi", "d/hi") => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "c") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - lfs_dir_open(&lfs, &dir, "d") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move check ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - lfs_dir_open(&lfs, &dir, "a/hi") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "b/hi") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "c/hi") => LFS_ERR_NOENT; - - lfs_dir_open(&lfs, &dir, "d/hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "bonjour") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hola") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "ohayo") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_dir_open(&lfs, &dir, "a/hello") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "b/hello") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "c/hello") => LFS_ERR_NOENT; - - lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, buffer, 5) => 5; - memcmp(buffer, "hola\n", 5) => 0; - lfs_file_read(&lfs, &file, buffer, 8) => 8; - memcmp(buffer, "bonjour\n", 8) => 0; - lfs_file_read(&lfs, &file, buffer, 6) => 6; - memcmp(buffer, "ohayo\n", 6) => 0; - lfs_file_close(&lfs, &file) => 0; - - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Move state stealing ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - lfs_remove(&lfs, "b") => 0; - lfs_remove(&lfs, "c") => 0; - - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - lfs_dir_open(&lfs, &dir, "a/hi") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "b") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "c") => LFS_ERR_NOENT; - - lfs_dir_open(&lfs, &dir, "d/hi") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "bonjour") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "hola") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "ohayo") => 0; - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_dir_open(&lfs, &dir, "a/hello") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "b") => LFS_ERR_NOENT; - lfs_dir_open(&lfs, &dir, "c") => LFS_ERR_NOENT; - - lfs_file_open(&lfs, &file, "d/hello", LFS_O_RDONLY) => 0; - lfs_file_read(&lfs, &file, buffer, 5) => 5; - memcmp(buffer, "hola\n", 5) => 0; - lfs_file_read(&lfs, &file, buffer, 8) => 8; - memcmp(buffer, "bonjour\n", 8) => 0; - lfs_file_read(&lfs, &file, buffer, 6) => 6; - memcmp(buffer, "ohayo\n", 6) => 0; - lfs_file_close(&lfs, &file) => 0; - - lfs_unmount(&lfs) => 0; -TEST - - -scripts/results.py diff --git a/tests_/test_move.toml b/tests/test_move.toml similarity index 100% rename from tests_/test_move.toml rename to tests/test_move.toml diff --git a/tests/test_orphan.sh b/tests/test_orphan.sh deleted file mode 100755 index b0a8493f..00000000 --- a/tests/test_orphan.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Orphan tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -echo "--- Orphan test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "parent") => 0; - lfs_mkdir(&lfs, "parent/orphan") => 0; - lfs_mkdir(&lfs, "parent/child") => 0; - lfs_remove(&lfs, "parent/orphan") => 0; -TEST -# corrupt most recent commit, this should be the update to the previous -# linked-list entry and should orphan the child -scripts/corrupt.py -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - - lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; - lfs_ssize_t before = lfs_fs_size(&lfs); - before => 8; - - lfs_unmount(&lfs) => 0; - lfs_mount(&lfs, &cfg) => 0; - - lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; - lfs_ssize_t orphaned = lfs_fs_size(&lfs); - orphaned => 8; - - lfs_mkdir(&lfs, "parent/otherchild") => 0; - - lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; - lfs_ssize_t deorphaned = lfs_fs_size(&lfs); - deorphaned => 8; - - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_orphans.toml b/tests/test_orphans.toml similarity index 100% rename from tests_/test_orphans.toml rename to tests/test_orphans.toml diff --git a/tests/test_paths.sh b/tests/test_paths.sh deleted file mode 100755 index cfdcd989..00000000 --- a/tests/test_paths.sh +++ /dev/null @@ -1,202 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Path tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "tea") => 0; - lfs_mkdir(&lfs, "coffee") => 0; - lfs_mkdir(&lfs, "soda") => 0; - lfs_mkdir(&lfs, "tea/hottea") => 0; - lfs_mkdir(&lfs, "tea/warmtea") => 0; - lfs_mkdir(&lfs, "tea/coldtea") => 0; - lfs_mkdir(&lfs, "coffee/hotcoffee") => 0; - lfs_mkdir(&lfs, "coffee/warmcoffee") => 0; - lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; - lfs_mkdir(&lfs, "soda/hotsoda") => 0; - lfs_mkdir(&lfs, "soda/warmsoda") => 0; - lfs_mkdir(&lfs, "soda/coldsoda") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Root path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "/tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - - lfs_mkdir(&lfs, "/milk1") => 0; - lfs_stat(&lfs, "/milk1", &info) => 0; - strcmp(info.name, "milk1") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Redundant slash path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "/tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "//tea//hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "///tea///hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - - lfs_mkdir(&lfs, "///milk2") => 0; - lfs_stat(&lfs, "///milk2", &info) => 0; - strcmp(info.name, "milk2") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Dot path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "./tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "/./tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "/././tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "/./tea/./hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - - lfs_mkdir(&lfs, "/./milk3") => 0; - lfs_stat(&lfs, "/./milk3", &info) => 0; - strcmp(info.name, "milk3") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Dot dot path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "coffee/../tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "tea/coldtea/../hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "coffee/coldcoffee/../../tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "coffee/../soda/../tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - - lfs_mkdir(&lfs, "coffee/../milk4") => 0; - lfs_stat(&lfs, "coffee/../milk4", &info) => 0; - strcmp(info.name, "milk4") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Trailing dot path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "tea/hottea/", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "tea/hottea/.", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "tea/hottea/./.", &info) => 0; - strcmp(info.name, "hottea") => 0; - lfs_stat(&lfs, "tea/hottea/..", &info) => 0; - strcmp(info.name, "tea") => 0; - lfs_stat(&lfs, "tea/hottea/../.", &info) => 0; - strcmp(info.name, "tea") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Root dot dot path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "coffee/../../../../../../tea/hottea", &info) => 0; - strcmp(info.name, "hottea") => 0; - - lfs_mkdir(&lfs, "coffee/../../../../../../milk5") => 0; - lfs_stat(&lfs, "coffee/../../../../../../milk5", &info) => 0; - strcmp(info.name, "milk5") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Root tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_stat(&lfs, "/", &info) => 0; - info.type => LFS_TYPE_DIR; - strcmp(info.name, "/") => 0; - - lfs_mkdir(&lfs, "/") => LFS_ERR_EXIST; - lfs_file_open(&lfs, &file, "/", LFS_O_WRONLY | LFS_O_CREAT) - => LFS_ERR_ISDIR; - - // more corner cases - lfs_remove(&lfs, "") => LFS_ERR_INVAL; - lfs_remove(&lfs, ".") => LFS_ERR_INVAL; - lfs_remove(&lfs, "..") => LFS_ERR_INVAL; - lfs_remove(&lfs, "/") => LFS_ERR_INVAL; - lfs_remove(&lfs, "//") => LFS_ERR_INVAL; - lfs_remove(&lfs, "./") => LFS_ERR_INVAL; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Sketchy path tests ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "dirt/ground") => LFS_ERR_NOENT; - lfs_mkdir(&lfs, "dirt/ground/earth") => LFS_ERR_NOENT; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Superblock conflict test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "littlefs") => 0; - lfs_remove(&lfs, "littlefs") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Max path test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(path, 'w', LFS_NAME_MAX+1); - path[LFS_NAME_MAX+2] = '\0'; - lfs_mkdir(&lfs, path) => LFS_ERR_NAMETOOLONG; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_NAMETOOLONG; - - memcpy(path, "coffee/", strlen("coffee/")); - memset(path+strlen("coffee/"), 'w', LFS_NAME_MAX+1); - path[strlen("coffee/")+LFS_NAME_MAX+2] = '\0'; - lfs_mkdir(&lfs, path) => LFS_ERR_NAMETOOLONG; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_NAMETOOLONG; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Really big path test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - memset(path, 'w', LFS_NAME_MAX); - path[LFS_NAME_MAX] = '\0'; - lfs_mkdir(&lfs, path) => 0; - lfs_remove(&lfs, path) => 0; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_remove(&lfs, path) => 0; - - memcpy(path, "coffee/", strlen("coffee/")); - memset(path+strlen("coffee/"), 'w', LFS_NAME_MAX); - path[strlen("coffee/")+LFS_NAME_MAX] = '\0'; - lfs_mkdir(&lfs, path) => 0; - lfs_remove(&lfs, path) => 0; - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT) => 0; - lfs_file_close(&lfs, &file) => 0; - lfs_remove(&lfs, path) => 0; - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_paths.toml b/tests/test_paths.toml similarity index 100% rename from tests_/test_paths.toml rename to tests/test_paths.toml diff --git a/tests/test_relocations.sh b/tests/test_relocations.sh deleted file mode 100755 index 5244e5ea..00000000 --- a/tests/test_relocations.sh +++ /dev/null @@ -1,139 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -ITERATIONS=20 -COUNT=10 - -echo "=== Relocation tests ===" -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - // fill up filesystem so only ~16 blocks are left - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "padding", LFS_O_CREAT | LFS_O_WRONLY) => 0; - memset(buffer, 0, 512); - while (LFS_BLOCK_COUNT - lfs_fs_size(&lfs) > 16) { - lfs_file_write(&lfs, &file, buffer, 512) => 512; - } - lfs_file_close(&lfs, &file) => 0; - // make a child dir to use in bounded space - lfs_mkdir(&lfs, "child") => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Dangling split dir test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - for (int j = 0; j < $ITERATIONS; j++) { - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; - lfs_file_close(&lfs, &file) => 0; - } - - lfs_dir_open(&lfs, &dir, "child") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "test%03d_loooooooooooooooooong_name", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - if (j == $ITERATIONS-1) { - break; - } - - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_remove(&lfs, path) => 0; - } - } - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "child") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "test%03d_loooooooooooooooooong_name", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_remove(&lfs, path) => 0; - } - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Outdated head test ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - for (int j = 0; j < $ITERATIONS; j++) { - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_file_open(&lfs, &file, path, LFS_O_CREAT | LFS_O_WRONLY) => 0; - lfs_file_close(&lfs, &file) => 0; - } - - lfs_dir_open(&lfs, &dir, "child") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "test%03d_loooooooooooooooooong_name", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.size => 0; - - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_file_open(&lfs, &file, path, LFS_O_WRONLY) => 0; - lfs_file_write(&lfs, &file, "hi", 2) => 2; - lfs_file_close(&lfs, &file) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - - lfs_dir_rewind(&lfs, &dir) => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "test%03d_loooooooooooooooooong_name", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.size => 2; - - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_file_open(&lfs, &file, path, LFS_O_WRONLY) => 0; - lfs_file_write(&lfs, &file, "hi", 2) => 2; - lfs_file_close(&lfs, &file) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - - lfs_dir_rewind(&lfs, &dir) => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - lfs_dir_read(&lfs, &dir, &info) => 1; - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "test%03d_loooooooooooooooooong_name", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - info.size => 2; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - for (int i = 0; i < $COUNT; i++) { - sprintf(path, "child/test%03d_loooooooooooooooooong_name", i); - lfs_remove(&lfs, path) => 0; - } - } - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_relocations.toml b/tests/test_relocations.toml similarity index 100% rename from tests_/test_relocations.toml rename to tests/test_relocations.toml diff --git a/tests/test_seek.sh b/tests/test_seek.sh deleted file mode 100755 index e136aa0d..00000000 --- a/tests/test_seek.sh +++ /dev/null @@ -1,505 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Seek tests ===" - -SMALLSIZE=4 -MEDIUMSIZE=128 -LARGESIZE=132 - -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "hello") => 0; - for (int i = 0; i < $LARGESIZE; i++) { - sprintf(path, "hello/kitty%03d", i); - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0; - - lfs_size_t size = strlen("kittycatcat"); - memcpy(buffer, "kittycatcat", size); - for (int j = 0; j < $LARGESIZE; j++) { - lfs_file_write(&lfs, &file, buffer, size); - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Simple dir seek ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - - lfs_soff_t pos; - int i; - for (i = 0; i < $SMALLSIZE; i++) { - sprintf(path, "kitty%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - pos = lfs_dir_tell(&lfs, &dir); - } - pos >= 0 => 1; - - lfs_dir_seek(&lfs, &dir, pos) => 0; - sprintf(path, "kitty%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - - lfs_dir_rewind(&lfs, &dir) => 0; - sprintf(path, "kitty%03d", 0); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - - lfs_dir_seek(&lfs, &dir, pos) => 0; - sprintf(path, "kitty%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Large dir seek ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_dir_open(&lfs, &dir, "hello") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - - lfs_soff_t pos; - int i; - for (i = 0; i < $MEDIUMSIZE; i++) { - sprintf(path, "kitty%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - pos = lfs_dir_tell(&lfs, &dir); - } - pos >= 0 => 1; - - lfs_dir_seek(&lfs, &dir, pos) => 0; - sprintf(path, "kitty%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - - lfs_dir_rewind(&lfs, &dir) => 0; - sprintf(path, "kitty%03d", 0); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, ".") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, "..") => 0; - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - - lfs_dir_seek(&lfs, &dir, pos) => 0; - sprintf(path, "kitty%03d", i); - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(info.name, path) => 0; - - lfs_dir_close(&lfs, &dir) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Simple file seek ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/kitty042", LFS_O_RDONLY) => 0; - - lfs_soff_t pos; - lfs_size_t size = strlen("kittycatcat"); - for (int i = 0; i < $SMALLSIZE; i++) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - pos = lfs_file_tell(&lfs, &file); - } - pos >= 0 => 1; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_rewind(&lfs, &file) => 0; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, size, LFS_SEEK_CUR) => 3*size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, -size, LFS_SEEK_CUR) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, -size, LFS_SEEK_END) >= 0 => 1; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - size = lfs_file_size(&lfs, &file); - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Large file seek ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/kitty042", LFS_O_RDONLY) => 0; - - lfs_soff_t pos; - lfs_size_t size = strlen("kittycatcat"); - for (int i = 0; i < $MEDIUMSIZE; i++) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - pos = lfs_file_tell(&lfs, &file); - } - pos >= 0 => 1; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_rewind(&lfs, &file) => 0; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, size, LFS_SEEK_CUR) => 3*size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, -size, LFS_SEEK_CUR) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, -size, LFS_SEEK_END) >= 0 => 1; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - size = lfs_file_size(&lfs, &file); - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Simple file seek and write ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/kitty042", LFS_O_RDWR) => 0; - - lfs_soff_t pos; - lfs_size_t size = strlen("kittycatcat"); - for (int i = 0; i < $SMALLSIZE; i++) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - pos = lfs_file_tell(&lfs, &file); - } - pos >= 0 => 1; - - memcpy(buffer, "doggodogdog", size); - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_write(&lfs, &file, buffer, size) => size; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "doggodogdog", size) => 0; - - lfs_file_rewind(&lfs, &file) => 0; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "doggodogdog", size) => 0; - - lfs_file_seek(&lfs, &file, -size, LFS_SEEK_END) >= 0 => 1; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - size = lfs_file_size(&lfs, &file); - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Large file seek and write ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/kitty042", LFS_O_RDWR) => 0; - - lfs_soff_t pos; - lfs_size_t size = strlen("kittycatcat"); - for (int i = 0; i < $MEDIUMSIZE; i++) { - lfs_file_read(&lfs, &file, buffer, size) => size; - if (i != $SMALLSIZE) { - memcmp(buffer, "kittycatcat", size) => 0; - } - pos = lfs_file_tell(&lfs, &file); - } - pos >= 0 => 1; - - memcpy(buffer, "doggodogdog", size); - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_write(&lfs, &file, buffer, size) => size; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "doggodogdog", size) => 0; - - lfs_file_rewind(&lfs, &file) => 0; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_seek(&lfs, &file, pos, LFS_SEEK_SET) => pos; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "doggodogdog", size) => 0; - - lfs_file_seek(&lfs, &file, -size, LFS_SEEK_END) >= 0 => 1; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - size = lfs_file_size(&lfs, &file); - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_CUR) => size; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Boundary seek and write ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/kitty042", LFS_O_RDWR) => 0; - - lfs_size_t size = strlen("hedgehoghog"); - const lfs_soff_t offsets[] = {512, 1020, 513, 1021, 511, 1019}; - - for (unsigned i = 0; i < sizeof(offsets) / sizeof(offsets[0]); i++) { - lfs_soff_t off = offsets[i]; - memcpy(buffer, "hedgehoghog", size); - lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off; - lfs_file_write(&lfs, &file, buffer, size) => size; - lfs_file_seek(&lfs, &file, off, LFS_SEEK_SET) => off; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "hedgehoghog", size) => 0; - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "kittycatcat", size) => 0; - - lfs_file_sync(&lfs, &file) => 0; - } - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Out-of-bounds seek ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/kitty042", LFS_O_RDWR) => 0; - - lfs_size_t size = strlen("kittycatcat"); - lfs_file_size(&lfs, &file) => $LARGESIZE*size; - lfs_file_seek(&lfs, &file, ($LARGESIZE+$SMALLSIZE)*size, - LFS_SEEK_SET) => ($LARGESIZE+$SMALLSIZE)*size; - lfs_file_read(&lfs, &file, buffer, size) => 0; - - memcpy(buffer, "porcupineee", size); - lfs_file_write(&lfs, &file, buffer, size) => size; - - lfs_file_seek(&lfs, &file, ($LARGESIZE+$SMALLSIZE)*size, - LFS_SEEK_SET) => ($LARGESIZE+$SMALLSIZE)*size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "porcupineee", size) => 0; - - lfs_file_seek(&lfs, &file, $LARGESIZE*size, - LFS_SEEK_SET) => $LARGESIZE*size; - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "\0\0\0\0\0\0\0\0\0\0\0", size) => 0; - - lfs_file_seek(&lfs, &file, -(($LARGESIZE+$SMALLSIZE)*size), - LFS_SEEK_CUR) => LFS_ERR_INVAL; - lfs_file_tell(&lfs, &file) => ($LARGESIZE+1)*size; - - lfs_file_seek(&lfs, &file, -(($LARGESIZE+2*$SMALLSIZE)*size), - LFS_SEEK_END) => LFS_ERR_INVAL; - lfs_file_tell(&lfs, &file) => ($LARGESIZE+1)*size; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Inline write and seek ---" -for SIZE in $SMALLSIZE $MEDIUMSIZE $LARGESIZE -do -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "hello/tinykitty$SIZE", - LFS_O_RDWR | LFS_O_CREAT) => 0; - int j = 0; - int k = 0; - - memcpy(buffer, "abcdefghijklmnopqrstuvwxyz", 26); - for (unsigned i = 0; i < $SIZE; i++) { - lfs_file_write(&lfs, &file, &buffer[j++ % 26], 1) => 1; - lfs_file_tell(&lfs, &file) => i+1; - lfs_file_size(&lfs, &file) => i+1; - } - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; - lfs_file_tell(&lfs, &file) => 0; - lfs_file_size(&lfs, &file) => $SIZE; - for (unsigned i = 0; i < $SIZE; i++) { - uint8_t c; - lfs_file_read(&lfs, &file, &c, 1) => 1; - c => buffer[k++ % 26]; - } - - lfs_file_sync(&lfs, &file) => 0; - lfs_file_tell(&lfs, &file) => $SIZE; - lfs_file_size(&lfs, &file) => $SIZE; - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; - for (unsigned i = 0; i < $SIZE; i++) { - lfs_file_write(&lfs, &file, &buffer[j++ % 26], 1) => 1; - lfs_file_tell(&lfs, &file) => i+1; - lfs_file_size(&lfs, &file) => $SIZE; - lfs_file_sync(&lfs, &file) => 0; - lfs_file_tell(&lfs, &file) => i+1; - lfs_file_size(&lfs, &file) => $SIZE; - if (i < $SIZE-2) { - uint8_t c[3]; - lfs_file_seek(&lfs, &file, -1, LFS_SEEK_CUR) => i; - lfs_file_read(&lfs, &file, &c, 3) => 3; - lfs_file_tell(&lfs, &file) => i+3; - lfs_file_size(&lfs, &file) => $SIZE; - lfs_file_seek(&lfs, &file, i+1, LFS_SEEK_SET) => i+1; - lfs_file_tell(&lfs, &file) => i+1; - lfs_file_size(&lfs, &file) => $SIZE; - } - } - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; - lfs_file_tell(&lfs, &file) => 0; - lfs_file_size(&lfs, &file) => $SIZE; - for (unsigned i = 0; i < $SIZE; i++) { - uint8_t c; - lfs_file_read(&lfs, &file, &c, 1) => 1; - c => buffer[k++ % 26]; - } - - lfs_file_sync(&lfs, &file) => 0; - lfs_file_tell(&lfs, &file) => $SIZE; - lfs_file_size(&lfs, &file) => $SIZE; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -done - -echo "--- Root seek test ---" -./scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - for (int i = 3; i < $MEDIUMSIZE; i++) { - sprintf(path, "hi%03d", i); - lfs_mkdir(&lfs, path) => 0; - } - - lfs_dir_open(&lfs, &dir, "/") => 0; - for (int i = 0; i < $MEDIUMSIZE; i++) { - if (i == 0) { - sprintf(path, "."); - } else if (i == 1) { - sprintf(path, ".."); - } else if (i == 2) { - sprintf(path, "hello"); - } else { - sprintf(path, "hi%03d", i); - } - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(path, info.name) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - for (int j = 0; j < $MEDIUMSIZE; j++) { - lfs_soff_t off = -1; - - lfs_dir_open(&lfs, &dir, "/") => 0; - for (int i = 0; i < $MEDIUMSIZE; i++) { - if (i == 0) { - sprintf(path, "."); - } else if (i == 1) { - sprintf(path, ".."); - } else if (i == 2) { - sprintf(path, "hello"); - } else { - sprintf(path, "hi%03d", i); - } - - if (i == j) { - off = lfs_dir_tell(&lfs, &dir); - off >= 0 => true; - } - - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(path, info.name) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - - lfs_dir_open(&lfs, &dir, "/") => 0; - lfs_dir_seek(&lfs, &dir, off) => 0; - for (int i = j; i < $MEDIUMSIZE; i++) { - if (i == 0) { - sprintf(path, "."); - } else if (i == 1) { - sprintf(path, ".."); - } else if (i == 2) { - sprintf(path, "hello"); - } else { - sprintf(path, "hi%03d", i); - } - - lfs_dir_read(&lfs, &dir, &info) => 1; - strcmp(path, info.name) => 0; - } - lfs_dir_read(&lfs, &dir, &info) => 0; - lfs_dir_close(&lfs, &dir) => 0; - } - - lfs_unmount(&lfs) => 0; -TEST - -scripts/results.py diff --git a/tests_/test_seek.toml b/tests/test_seek.toml similarity index 100% rename from tests_/test_seek.toml rename to tests/test_seek.toml diff --git a/tests/test_truncate.sh b/tests/test_truncate.sh deleted file mode 100755 index f33717df..00000000 --- a/tests/test_truncate.sh +++ /dev/null @@ -1,355 +0,0 @@ -#!/bin/bash -set -eu -export TEST_FILE=$0 -trap 'export TEST_LINE=$LINENO' DEBUG - -echo "=== Truncate tests ===" - -SMALLSIZE=32 -MEDIUMSIZE=2048 -LARGESIZE=8192 - -rm -rf blocks -scripts/test.py << TEST - lfs_format(&lfs, &cfg) => 0; -TEST - -echo "--- Simple truncate ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldynoop", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - - strcpy((char*)buffer, "hair"); - lfs_size_t size = strlen((char*)buffer); - for (lfs_off_t j = 0; j < $LARGESIZE; j += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_size(&lfs, &file) => $LARGESIZE; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldynoop", LFS_O_RDWR) => 0; - lfs_file_size(&lfs, &file) => $LARGESIZE; - - lfs_file_truncate(&lfs, &file, $MEDIUMSIZE) => 0; - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldynoop", LFS_O_RDONLY) => 0; - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - lfs_size_t size = strlen("hair"); - for (lfs_off_t j = 0; j < $MEDIUMSIZE; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "hair", size) => 0; - } - lfs_file_read(&lfs, &file, buffer, size) => 0; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Truncate and read ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldyread", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - - strcpy((char*)buffer, "hair"); - lfs_size_t size = strlen((char*)buffer); - for (lfs_off_t j = 0; j < $LARGESIZE; j += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_size(&lfs, &file) => $LARGESIZE; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldyread", LFS_O_RDWR) => 0; - lfs_file_size(&lfs, &file) => $LARGESIZE; - - lfs_file_truncate(&lfs, &file, $MEDIUMSIZE) => 0; - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - lfs_size_t size = strlen("hair"); - for (lfs_off_t j = 0; j < $MEDIUMSIZE; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "hair", size) => 0; - } - lfs_file_read(&lfs, &file, buffer, size) => 0; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldyread", LFS_O_RDONLY) => 0; - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - lfs_size_t size = strlen("hair"); - for (lfs_off_t j = 0; j < $MEDIUMSIZE; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "hair", size) => 0; - } - lfs_file_read(&lfs, &file, buffer, size) => 0; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Write, truncate, and read ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "sequence", - LFS_O_RDWR | LFS_O_CREAT | LFS_O_TRUNC) => 0; - - lfs_size_t size = lfs.cfg->cache_size; - lfs_size_t qsize = size / 4; - uint8_t *wb = buffer; - uint8_t *rb = buffer + size; - for (lfs_off_t j = 0; j < size; ++j) { - wb[j] = j; - } - - /* Spread sequence over size */ - lfs_file_write(&lfs, &file, wb, size) => size; - lfs_file_size(&lfs, &file) => size; - lfs_file_tell(&lfs, &file) => size; - - lfs_file_seek(&lfs, &file, 0, LFS_SEEK_SET) => 0; - lfs_file_tell(&lfs, &file) => 0; - - /* Chop off the last quarter */ - lfs_size_t trunc = size - qsize; - lfs_file_truncate(&lfs, &file, trunc) => 0; - lfs_file_tell(&lfs, &file) => 0; - lfs_file_size(&lfs, &file) => trunc; - - /* Read should produce first 3/4 */ - lfs_file_read(&lfs, &file, rb, size) => trunc; - memcmp(rb, wb, trunc) => 0; - - /* Move to 1/4 */ - lfs_file_size(&lfs, &file) => trunc; - lfs_file_seek(&lfs, &file, qsize, LFS_SEEK_SET) => qsize; - lfs_file_tell(&lfs, &file) => qsize; - - /* Chop to 1/2 */ - trunc -= qsize; - lfs_file_truncate(&lfs, &file, trunc) => 0; - lfs_file_tell(&lfs, &file) => qsize; - lfs_file_size(&lfs, &file) => trunc; - - /* Read should produce second quarter */ - lfs_file_read(&lfs, &file, rb, size) => trunc - qsize; - memcmp(rb, wb + qsize, trunc - qsize) => 0; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -echo "--- Truncate and write ---" -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldywrite", - LFS_O_WRONLY | LFS_O_CREAT) => 0; - - strcpy((char*)buffer, "hair"); - lfs_size_t size = strlen((char*)buffer); - for (lfs_off_t j = 0; j < $LARGESIZE; j += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_size(&lfs, &file) => $LARGESIZE; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldywrite", LFS_O_RDWR) => 0; - lfs_file_size(&lfs, &file) => $LARGESIZE; - - lfs_file_truncate(&lfs, &file, $MEDIUMSIZE) => 0; - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - strcpy((char*)buffer, "bald"); - lfs_size_t size = strlen((char*)buffer); - for (lfs_off_t j = 0; j < $MEDIUMSIZE; j += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - lfs_mount(&lfs, &cfg) => 0; - lfs_file_open(&lfs, &file, "baldywrite", LFS_O_RDONLY) => 0; - lfs_file_size(&lfs, &file) => $MEDIUMSIZE; - - lfs_size_t size = strlen("bald"); - for (lfs_off_t j = 0; j < $MEDIUMSIZE; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "bald", size) => 0; - } - lfs_file_read(&lfs, &file, buffer, size) => 0; - - lfs_file_close(&lfs, &file) => 0; - lfs_unmount(&lfs) => 0; -TEST - -# More aggressive general truncation tests -truncate_test() { -STARTSIZES="$1" -STARTSEEKS="$2" -HOTSIZES="$3" -COLDSIZES="$4" -scripts/test.py << TEST - static const lfs_off_t startsizes[] = {$STARTSIZES}; - static const lfs_off_t startseeks[] = {$STARTSEEKS}; - static const lfs_off_t hotsizes[] = {$HOTSIZES}; - - lfs_mount(&lfs, &cfg) => 0; - - for (unsigned i = 0; i < sizeof(startsizes)/sizeof(startsizes[0]); i++) { - sprintf(path, "hairyhead%d", i); - lfs_file_open(&lfs, &file, path, - LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; - - strcpy((char*)buffer, "hair"); - lfs_size_t size = strlen((char*)buffer); - for (lfs_off_t j = 0; j < startsizes[i]; j += size) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - lfs_file_size(&lfs, &file) => startsizes[i]; - - if (startseeks[i] != startsizes[i]) { - lfs_file_seek(&lfs, &file, - startseeks[i], LFS_SEEK_SET) => startseeks[i]; - } - - lfs_file_truncate(&lfs, &file, hotsizes[i]) => 0; - lfs_file_size(&lfs, &file) => hotsizes[i]; - - lfs_file_close(&lfs, &file) => 0; - } - - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - static const lfs_off_t startsizes[] = {$STARTSIZES}; - static const lfs_off_t hotsizes[] = {$HOTSIZES}; - static const lfs_off_t coldsizes[] = {$COLDSIZES}; - - lfs_mount(&lfs, &cfg) => 0; - - for (unsigned i = 0; i < sizeof(startsizes)/sizeof(startsizes[0]); i++) { - sprintf(path, "hairyhead%d", i); - lfs_file_open(&lfs, &file, path, LFS_O_RDWR) => 0; - lfs_file_size(&lfs, &file) => hotsizes[i]; - - lfs_size_t size = strlen("hair"); - lfs_off_t j = 0; - for (; j < startsizes[i] && j < hotsizes[i]; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "hair", size) => 0; - } - - for (; j < hotsizes[i]; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "\0\0\0\0", size) => 0; - } - - lfs_file_truncate(&lfs, &file, coldsizes[i]) => 0; - lfs_file_size(&lfs, &file) => coldsizes[i]; - - lfs_file_close(&lfs, &file) => 0; - } - - lfs_unmount(&lfs) => 0; -TEST -scripts/test.py << TEST - static const lfs_off_t startsizes[] = {$STARTSIZES}; - static const lfs_off_t hotsizes[] = {$HOTSIZES}; - static const lfs_off_t coldsizes[] = {$COLDSIZES}; - - lfs_mount(&lfs, &cfg) => 0; - - for (unsigned i = 0; i < sizeof(startsizes)/sizeof(startsizes[0]); i++) { - sprintf(path, "hairyhead%d", i); - lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; - lfs_file_size(&lfs, &file) => coldsizes[i]; - - lfs_size_t size = strlen("hair"); - lfs_off_t j = 0; - for (; j < startsizes[i] && j < hotsizes[i] && j < coldsizes[i]; - j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "hair", size) => 0; - } - - for (; j < coldsizes[i]; j += size) { - lfs_file_read(&lfs, &file, buffer, size) => size; - memcmp(buffer, "\0\0\0\0", size) => 0; - } - - lfs_file_close(&lfs, &file) => 0; - } - - lfs_unmount(&lfs) => 0; -TEST -} - -echo "--- Cold shrinking truncate ---" -truncate_test \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" - -echo "--- Cold expanding truncate ---" -truncate_test \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" - -echo "--- Warm shrinking truncate ---" -truncate_test \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - " 0, 0, 0, 0, 0" - -echo "--- Warm expanding truncate ---" -truncate_test \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" - -echo "--- Mid-file shrinking truncate ---" -truncate_test \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - " $LARGESIZE, $LARGESIZE, $LARGESIZE, $LARGESIZE, $LARGESIZE" \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - " 0, 0, 0, 0, 0" - -echo "--- Mid-file expanding truncate ---" -truncate_test \ - " 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE, 2*$LARGESIZE" \ - " 0, 0, $SMALLSIZE, $MEDIUMSIZE, $LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" \ - "2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE, 2*$LARGESIZE" - -scripts/results.py diff --git a/tests_/test_truncate.toml b/tests/test_truncate.toml similarity index 100% rename from tests_/test_truncate.toml rename to tests/test_truncate.toml From 517d3414c5e04eedb07be2e58107c1f96b8b8684 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Wed, 29 Jan 2020 01:45:19 -0600 Subject: [PATCH 16/41] Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet. --- Makefile | 2 +- lfs.c | 131 +++++++++++++++++++++++------------- scripts/readtree.py | 23 ++++++- scripts/test.py | 93 ++++++++++++++++--------- tests/test_alloc.toml | 8 +-- tests/test_dirs.toml | 2 +- tests/test_exhaustion.toml | 22 +++--- tests/test_orphans.toml | 12 ++-- tests/test_relocations.toml | 16 +++-- 9 files changed, 200 insertions(+), 109 deletions(-) diff --git a/Makefile b/Makefile index 40245b8f..7cc73443 100644 --- a/Makefile +++ b/Makefile @@ -45,7 +45,7 @@ test: ./scripts/test.py $(TFLAGS) .SECONDEXPANSION: test%: tests/test$$(firstword $$(subst \#, ,%)).toml - ./scripts/test.py $(TFLAGS) $@ + ./scripts/test.py $@ $(TFLAGS) -include $(DEP) diff --git a/lfs.c b/lfs.c index bc6e1bc6..0a27e9be 100644 --- a/lfs.c +++ b/lfs.c @@ -804,8 +804,11 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, // next commit not yet programmed or we're not in valid range if (!lfs_tag_isvalid(tag) || off + lfs_tag_dsize(tag) > lfs->cfg->block_size) { + //printf("read block %d valid %d ntag %08x ptag %08x off %d (off %d dsize %dblock %d)\n", dir->pair[0], lfs_tag_isvalid(tag), tag, ptag, dir->off, off, lfs_tag_dsize(tag), lfs->cfg->block_size); + //printf("read block %d erased = %d\n", dir->pair[0], (lfs_tag_type1(ptag) == LFS_TYPE_CRC && dir->off % lfs->cfg->prog_size == 0)); dir->erased = (lfs_tag_type1(ptag) == LFS_TYPE_CRC && - dir->off % lfs->cfg->prog_size == 0); + dir->off % lfs->cfg->prog_size == 0 && + !lfs_tag_isvalid(tag)); break; } @@ -1230,6 +1233,7 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { const lfs_off_t off1 = commit->off + sizeof(lfs_tag_t); const lfs_off_t end = lfs_alignup(off1 + sizeof(uint32_t), lfs->cfg->prog_size); + uint32_t ncrc = commit->crc; // create crc tags to fill up remainder of commit, note that // padding is not crcd, which lets fetches skip padding but @@ -1266,6 +1270,7 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { return err; } + ncrc = commit->crc; commit->off += sizeof(tag)+lfs_tag_size(tag); commit->ptag = tag ^ ((lfs_tag_t)reset << 31); commit->crc = LFS_BLOCK_NULL; // reset crc for next "commit" @@ -1292,6 +1297,13 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { return err; } + // check against written crc to detect if block is readonly + // (we may pick up old commits) +// TODO rm me? +// if (i == noff && crc != ncrc) { +// return LFS_ERR_CORRUPT; +// } + crc = lfs_crc(crc, &dat, 1); } @@ -1320,6 +1332,9 @@ static int lfs_dir_alloc(lfs_t *lfs, lfs_mdir_t *dir) { } } + // zero for reproducability in case initial block is unreadable + dir->rev = 0; + // rather than clobbering one of the blocks we just pretend // the revision may be valid int err = lfs_bd_read(lfs, @@ -1385,6 +1400,7 @@ static int lfs_dir_split(lfs_t *lfs, return err; } + //dir->rev += 1; // TODO really? dir->tail[0] = tail.pair[0]; dir->tail[1] = tail.pair[1]; dir->split = true; @@ -1420,9 +1436,9 @@ static int lfs_dir_compact(lfs_t *lfs, lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount, lfs_mdir_t *source, uint16_t begin, uint16_t end) { // save some state in case block is bad - const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]}; + const lfs_block_t oldpair[2] = {dir->pair[0], dir->pair[1]}; bool relocated = false; - bool exhausted = false; + bool tired = false; // should we split? while (end - begin > 1) { @@ -1468,9 +1484,9 @@ static int lfs_dir_compact(lfs_t *lfs, } // increment revision count - dir->rev += 1; + uint32_t nrev = dir->rev + 1; if (lfs->cfg->block_cycles > 0 && - (dir->rev % (lfs->cfg->block_cycles+1) == 0)) { + (nrev % (lfs->cfg->block_cycles+1) == 0)) { if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) { // oh no! we're writing too much to the superblock, // should we expand? @@ -1482,7 +1498,8 @@ static int lfs_dir_compact(lfs_t *lfs, // do we have extra space? littlefs can't reclaim this space // by itself, so expand cautiously if ((lfs_size_t)res < lfs->cfg->block_count/2) { - LFS_DEBUG("Expanding superblock at rev %"PRIu32, dir->rev); + LFS_DEBUG("Expanding superblock at rev %"PRIu32, nrev); + //dir->rev += 1; // TODO hmm int err = lfs_dir_split(lfs, dir, attrs, attrcount, source, begin, end); if (err && err != LFS_ERR_NOSPC) { @@ -1506,7 +1523,7 @@ static int lfs_dir_compact(lfs_t *lfs, #endif } else { // we're writing too much, time to relocate - exhausted = true; + tired = true; goto relocate; } } @@ -1535,10 +1552,10 @@ static int lfs_dir_compact(lfs_t *lfs, } // write out header - dir->rev = lfs_tole32(dir->rev); + nrev = lfs_tole32(nrev); err = lfs_dir_commitprog(lfs, &commit, - &dir->rev, sizeof(dir->rev)); - dir->rev = lfs_fromle32(dir->rev); + &nrev, sizeof(nrev)); + nrev = lfs_fromle32(nrev); if (err) { if (err == LFS_ERR_CORRUPT) { goto relocate; @@ -1612,17 +1629,35 @@ static int lfs_dir_compact(lfs_t *lfs, return err; } - // successful compaction, swap dir pair to indicate most recent + // TODO huh? LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0); - lfs_pair_swap(dir->pair); - dir->count = end - begin; - dir->off = commit.off; - dir->etag = commit.ptag; // update gstate lfs->gdelta = (lfs_gstate_t){0}; if (!relocated) { lfs->gdisk = lfs->gstate; } + + // TODO here?? + if (relocated) { + // update references if we relocated + LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + err = lfs_fs_relocate(lfs, oldpair, dir->pair); + if (err) { + // TODO make better + dir->pair[1] = oldpair[1]; // + return err; + } + LFS_DEBUG("Relocated %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + } + + // successful compaction, swap dir pair to indicate most recent + lfs_pair_swap(dir->pair); + dir->rev = nrev; + dir->count = end - begin; + dir->off = commit.off; + dir->etag = commit.ptag; } break; @@ -1630,36 +1665,26 @@ static int lfs_dir_compact(lfs_t *lfs, // commit was corrupted, drop caches and prepare to relocate block relocated = true; lfs_cache_drop(lfs, &lfs->pcache); - if (!exhausted) { + if (!tired) { LFS_DEBUG("Bad block at %"PRIx32, dir->pair[1]); } // can't relocate superblock, filesystem is now frozen - if (lfs_pair_cmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) { - LFS_WARN("Superblock %"PRIx32" has become unwritable", oldpair[1]); + if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) { + LFS_WARN("Superblock %"PRIx32" has become unwritable", dir->pair[1]); return LFS_ERR_NOSPC; } // relocate half of pair int err = lfs_alloc(lfs, &dir->pair[1]); - if (err && (err != LFS_ERR_NOSPC || !exhausted)) { + if (err && (err != LFS_ERR_NOSPC || !tired)) { return err; } - exhausted = false; + tired = false; continue; } - if (relocated) { - // update references if we relocated - LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, - oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - int err = lfs_fs_relocate(lfs, oldpair, dir->pair); - if (err) { - return err; - } - } - return 0; } @@ -2204,16 +2229,16 @@ static int lfs_ctz_extend(lfs_t *lfs, return 0; } - size -= 1; - lfs_off_t index = lfs_ctz_index(lfs, &size); - size += 1; + lfs_size_t noff = size - 1; + lfs_off_t index = lfs_ctz_index(lfs, &noff); + noff = noff + 1; // just copy out the last block if it is incomplete - if (size != lfs->cfg->block_size) { - for (lfs_off_t i = 0; i < size; i++) { + if (noff != lfs->cfg->block_size) { + for (lfs_off_t i = 0; i < noff; i++) { uint8_t data; err = lfs_bd_read(lfs, - NULL, rcache, size-i, + NULL, rcache, noff-i, head, i, &data, 1); if (err) { return err; @@ -2231,19 +2256,19 @@ static int lfs_ctz_extend(lfs_t *lfs, } *block = nblock; - *off = size; + *off = noff; return 0; } // append block index += 1; lfs_size_t skips = lfs_ctz(index) + 1; - + lfs_block_t nhead = head; for (lfs_off_t i = 0; i < skips; i++) { - head = lfs_tole32(head); + nhead = lfs_tole32(nhead); err = lfs_bd_prog(lfs, pcache, rcache, true, - nblock, 4*i, &head, 4); - head = lfs_fromle32(head); + nblock, 4*i, &nhead, 4); + nhead = lfs_fromle32(nhead); if (err) { if (err == LFS_ERR_CORRUPT) { goto relocate; @@ -2253,15 +2278,15 @@ static int lfs_ctz_extend(lfs_t *lfs, if (i != skips-1) { err = lfs_bd_read(lfs, - NULL, rcache, sizeof(head), - head, 4*i, &head, sizeof(head)); - head = lfs_fromle32(head); + NULL, rcache, sizeof(nhead), + nhead, 4*i, &nhead, sizeof(nhead)); + nhead = lfs_fromle32(nhead); if (err) { return err; } } - LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count); + LFS_ASSERT(nhead >= 2 && nhead <= lfs->cfg->block_count); } *block = nblock; @@ -3821,6 +3846,17 @@ int lfs_fs_traverse(lfs_t *lfs, LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } + } else if (lfs_gstate_hasorphans(&lfs->gstate) && + lfs_tag_type3(tag) == LFS_TYPE_DIRSTRUCT) { + // TODO HMMMMMM HMMMMMMMMMMMMMMMMMMM + for (int i = 0; i < 2; i++) { + //printf("HMM %x\n", (&ctz.head)[i]); + err = cb(data, (&ctz.head)[i]); + if (err) { + LFS_TRACE("lfs_fs_traverse -> %d", err); + return err; + } + } } } } @@ -3902,7 +3938,9 @@ static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2], // use fetchmatch with callback to find pairs parent->tail[0] = 0; parent->tail[1] = 1; + int i = 0; while (!lfs_pair_isnull(parent->tail)) { + i += 1; lfs_stag_t tag = lfs_dir_fetchmatch(lfs, parent, parent->tail, LFS_MKTAG(0x7ff, 0, 0x3ff), LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, 8), @@ -3910,6 +3948,7 @@ static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2], lfs_fs_parent_match, &(struct lfs_fs_parent_match){ lfs, {pair[0], pair[1]}}); if (tag && tag != LFS_ERR_NOENT) { + //printf("PARENT %d\n", i); return tag; } } @@ -3966,6 +4005,7 @@ static int lfs_fs_relocate(lfs_t *lfs, } } + //printf("parent %x %x\n", parent.pair[0], parent.pair[1]); lfs_pair_tole32(newpair); int err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS( {LFS_MKTAG_IF(moveid != 0x3ff, @@ -4000,6 +4040,7 @@ static int lfs_fs_relocate(lfs_t *lfs, } // replace bad pair, either we clean up desync, or no desync occured + //printf("pred %x %x\n", parent.pair[0], parent.pair[1]); lfs_pair_tole32(newpair); err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS( {LFS_MKTAG_IF(moveid != 0x3ff, diff --git a/scripts/readtree.py b/scripts/readtree.py index 30e3cfc0..ecfdab94 100755 --- a/scripts/readtree.py +++ b/scripts/readtree.py @@ -118,9 +118,17 @@ def main(args): superblock = None gstate = b'' mdirs = [] + cycle = False tail = (args.block1, args.block2) hard = False while True: + for m in it.chain((m for d in dirs for m in d), mdirs): + if set(m.blocks) == set(tail): + # cycle detected + cycle = m.blocks + if cycle: + break + # load mdir data = [] blocks = {} @@ -129,6 +137,7 @@ def main(args): data.append(f.read(args.block_size) .ljust(args.block_size, b'\xff')) blocks[id(data[-1])] = block + mdir = MetadataPair(data) mdir.blocks = tuple(blocks[id(p.data)] for p in mdir.pair) @@ -171,7 +180,7 @@ def main(args): # find paths dirtable = {} for dir in dirs: - dirtable[tuple(sorted(dir[0].blocks))] = dir + dirtable[frozenset(dir[0].blocks)] = dir pending = [("/", dirs[0])] while pending: @@ -183,7 +192,7 @@ def main(args): npath = tag.data.decode('utf8') dirstruct = mdir[Tag('dirstruct', tag.id, 0)] nblocks = struct.unpack(' {%#x, %#x} ***" % (cycle[0], cycle[1])) + + if cycle: + return 2 + elif not all(mdir for dir in dirs for mdir in dir): + return 1 + else: + return 0; if __name__ == "__main__": import argparse diff --git a/scripts/test.py b/scripts/test.py index 02801f8c..3c3d692d 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -182,7 +182,23 @@ def shouldtest(self, **args): elif args.get('no_internal', False) and self.in_ is not None: return False elif self.if_ is not None: - return eval(self.if_, None, self.defines.copy()) + if_ = self.if_ + print(if_) + while True: + for k, v in self.defines.items(): + if k in if_: + if_ = if_.replace(k, '(%s)' % v) + print(if_) + break + else: + break + if_ = ( + re.sub('(\&\&|\?)', ' and ', + re.sub('(\|\||:)', ' or ', + re.sub('!(?!=)', ' not ', if_)))) + print(if_) + print('---', eval(if_), '---') + return eval(if_) else: return True @@ -235,33 +251,37 @@ def test(self, exec=[], persist=False, cycles=None, mpty = os.fdopen(mpty, 'r', 1) stdout = [] assert_ = None - while True: - try: - line = mpty.readline() - except OSError as e: - if e.errno == errno.EIO: - break - raise - stdout.append(line) - if args.get('verbose', False): - sys.stdout.write(line) - # intercept asserts - m = re.match( - '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' - .format('(?:\033\[[\d;]*.| )*', 'assert'), - line) - if m and assert_ is None: + try: + while True: try: - with open(m.group(1)) as f: - lineno = int(m.group(2)) - line = next(it.islice(f, lineno-1, None)).strip('\n') - assert_ = { - 'path': m.group(1), - 'line': line, - 'lineno': lineno, - 'message': m.group(3)} - except: - pass + line = mpty.readline() + except OSError as e: + if e.errno == errno.EIO: + break + raise + stdout.append(line) + if args.get('verbose', False): + sys.stdout.write(line) + # intercept asserts + m = re.match( + '^{0}([^:]+):(\d+):(?:\d+:)?{0}{1}:{0}(.*)$' + .format('(?:\033\[[\d;]*.| )*', 'assert'), + line) + if m and assert_ is None: + try: + with open(m.group(1)) as f: + lineno = int(m.group(2)) + line = (next(it.islice(f, lineno-1, None)) + .strip('\n')) + assert_ = { + 'path': m.group(1), + 'line': line, + 'lineno': lineno, + 'message': m.group(3)} + except: + pass + except KeyboardInterrupt: + raise TestFailure(self, 1, stdout, None) proc.wait() # did we pass? @@ -654,6 +674,10 @@ def main(**args): if filtered != sum(len(suite.perms) for suite in suites): print('filtered down to %d permutations' % filtered) + # only requested to build? + if args.get('build', False): + return 0 + print('====== testing ======') try: for suite in suites: @@ -678,18 +702,19 @@ def main(**args): perm=perm, path=perm.suite.path, lineno=perm.lineno, returncode=perm.result.returncode or 0)) if perm.result.stdout: - for line in (perm.result.stdout - if not perm.result.assert_ - else perm.result.stdout[:-1]): + if perm.result.assert_: + stdout = perm.result.stdout[:-1] + else: + stdout = perm.result.stdout + if (not args.get('verbose', False) and len(stdout) > 5): + sys.stdout.write('...\n') + for line in stdout[-5:]: sys.stdout.write(line) if perm.result.assert_: sys.stdout.write( "\033[01m{path}:{lineno}:\033[01;31massert:\033[m " "{message}\n{line}\n".format( **perm.result.assert_)) - else: - for line in perm.result.stdout: - sys.stdout.write(line) sys.stdout.write('\n') failed += 1 @@ -728,6 +753,8 @@ def main(**args): parser.add_argument('-p', '--persist', choices=['erase', 'noerase'], nargs='?', const='erase', help="Store disk image in a file.") + parser.add_argument('-b', '--build', action='store_true', + help="Only build the tests, do not execute.") parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], nargs='?', const='assert', help="Drop into gdb on test failure.") diff --git a/tests/test_alloc.toml b/tests/test_alloc.toml index 8bfb5edc..f15a7461 100644 --- a/tests/test_alloc.toml +++ b/tests/test_alloc.toml @@ -329,7 +329,7 @@ code = ''' [[case]] # chained dir exhaustion test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 -if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' +if = 'LFS_BLOCK_SIZE == 512 && LFS_BLOCK_COUNT == 1024' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -400,7 +400,7 @@ code = ''' [[case]] # split dir test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 -if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' +if = 'LFS_BLOCK_SIZE == 512 && LFS_BLOCK_COUNT == 1024' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -445,7 +445,7 @@ code = ''' [[case]] # outdated lookahead test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 -if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' +if = 'LFS_BLOCK_SIZE == 512 && LFS_BLOCK_COUNT == 1024' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -510,7 +510,7 @@ code = ''' [[case]] # outdated lookahead and split dir test define.LFS_BLOCK_SIZE = 512 define.LFS_BLOCK_COUNT = 1024 -if = 'LFS_BLOCK_SIZE == 512 and LFS_BLOCK_COUNT == 1024' +if = 'LFS_BLOCK_SIZE == 512 && LFS_BLOCK_COUNT == 1024' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; diff --git a/tests/test_dirs.toml b/tests/test_dirs.toml index 712f24ff..f3a819d1 100644 --- a/tests/test_dirs.toml +++ b/tests/test_dirs.toml @@ -155,7 +155,7 @@ code = ''' ''' [[case]] # reentrant many directory creation/rename/removal -define.N = [5, 25] +define.N = [5, 10] # TODO changed from 20, should we be able to do more? reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); diff --git a/tests/test_exhaustion.toml b/tests/test_exhaustion.toml index fe64a806..7faa1764 100644 --- a/tests/test_exhaustion.toml +++ b/tests/test_exhaustion.toml @@ -158,7 +158,7 @@ exhausted: # check for. [[case]] # wear-level test running a filesystem to exhaustion -define.LFS_ERASE_CYCLES = 10 +define.LFS_ERASE_CYCLES = 20 define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' define.LFS_BADBLOCK_BEHAVIOR = [ @@ -168,11 +168,6 @@ define.LFS_BADBLOCK_BEHAVIOR = [ ] define.FILES = 10 code = ''' - lfs_format(&lfs, &cfg) => 0; - lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "roadrunner") => 0; - lfs_unmount(&lfs) => 0; - uint32_t run_cycles[2]; const uint32_t run_block_count[2] = {LFS_BLOCK_COUNT/2, LFS_BLOCK_COUNT}; @@ -182,6 +177,11 @@ code = ''' (b < run_block_count[run]) ? 0 : LFS_ERASE_CYCLES) => 0; } + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "roadrunner") => 0; + lfs_unmount(&lfs) => 0; + uint32_t cycle = 0; while (true) { lfs_mount(&lfs, &cfg) => 0; @@ -247,11 +247,11 @@ exhausted: } // check we increased the lifetime by 2x with ~10% error - LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/10); + LFS_ASSERT(run_cycles[1]*110/100 > 2*run_cycles[0]); ''' [[case]] # wear-level test + expanding superblock -define.LFS_ERASE_CYCLES = 10 +define.LFS_ERASE_CYCLES = 20 define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' define.LFS_BADBLOCK_BEHAVIOR = [ @@ -261,8 +261,6 @@ define.LFS_BADBLOCK_BEHAVIOR = [ ] define.FILES = 10 code = ''' - lfs_format(&lfs, &cfg) => 0; - uint32_t run_cycles[2]; const uint32_t run_block_count[2] = {LFS_BLOCK_COUNT/2, LFS_BLOCK_COUNT}; @@ -272,6 +270,8 @@ code = ''' (b < run_block_count[run]) ? 0 : LFS_ERASE_CYCLES) => 0; } + lfs_format(&lfs, &cfg) => 0; + uint32_t cycle = 0; while (true) { lfs_mount(&lfs, &cfg) => 0; @@ -337,5 +337,5 @@ exhausted: } // check we increased the lifetime by 2x with ~10% error - LFS_ASSERT(run_cycles[1] > 2*run_cycles[0]-run_cycles[0]/10); + LFS_ASSERT(run_cycles[1]*110/100 > 2*run_cycles[0]); ''' diff --git a/tests/test_orphans.toml b/tests/test_orphans.toml index 3bf0454e..ea4ca507 100644 --- a/tests/test_orphans.toml +++ b/tests/test_orphans.toml @@ -31,13 +31,13 @@ code = ''' lfs_mount(&lfs, &cfg) => 0; lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; lfs_stat(&lfs, "parent/child", &info) => 0; - lfs_fs_size(&lfs) => 8; + lfs_fs_size(&lfs) => 12; lfs_unmount(&lfs) => 0; lfs_mount(&lfs, &cfg) => 0; lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; lfs_stat(&lfs, "parent/child", &info) => 0; - lfs_fs_size(&lfs) => 8; + lfs_fs_size(&lfs) => 12; // this mkdir should both create a dir and deorphan, so size // should be unchanged lfs_mkdir(&lfs, "parent/otherchild") => 0; @@ -57,10 +57,12 @@ code = ''' [[case]] # reentrant testing for orphans, basically just spam mkdir/remove reentrant = true +# TODO fix this case, caused by non-DAG trees +if = '!(DEPTH == 3 && LFS_CACHE_SIZE != 64)' define = [ - {FILES=6, DEPTH=1, CYCLES=50}, - {FILES=26, DEPTH=1, CYCLES=50}, - {FILES=3, DEPTH=3, CYCLES=50}, + {FILES=6, DEPTH=1, CYCLES=20}, + {FILES=26, DEPTH=1, CYCLES=20}, + {FILES=3, DEPTH=3, CYCLES=20}, ] code = ''' err = lfs_mount(&lfs, &cfg); diff --git a/tests/test_relocations.toml b/tests/test_relocations.toml index ad267b51..71b10475 100644 --- a/tests/test_relocations.toml +++ b/tests/test_relocations.toml @@ -147,10 +147,12 @@ code = ''' # orphan testing, except here we also set block_cycles so that # almost every tree operation needs a relocation reentrant = true +# TODO fix this case, caused by non-DAG trees +if = '!(DEPTH == 3 && LFS_CACHE_SIZE != 64)' define = [ - {FILES=6, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, - {FILES=26, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, - {FILES=3, DEPTH=3, CYCLES=50, LFS_BLOCK_CYCLES=1}, + {FILES=6, DEPTH=1, CYCLES=20, LFS_BLOCK_CYCLES=1}, + {FILES=26, DEPTH=1, CYCLES=20, LFS_BLOCK_CYCLES=1}, + {FILES=3, DEPTH=3, CYCLES=20, LFS_BLOCK_CYCLES=1}, ] code = ''' err = lfs_mount(&lfs, &cfg); @@ -207,10 +209,12 @@ code = ''' [[case]] # reentrant testing for relocations, but now with random renames! reentrant = true +# TODO fix this case, caused by non-DAG trees +if = '!(DEPTH == 3 && LFS_CACHE_SIZE != 64)' define = [ - {FILES=6, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, - {FILES=26, DEPTH=1, CYCLES=50, LFS_BLOCK_CYCLES=1}, - {FILES=3, DEPTH=3, CYCLES=50, LFS_BLOCK_CYCLES=1}, + {FILES=6, DEPTH=1, CYCLES=20, LFS_BLOCK_CYCLES=1}, + {FILES=26, DEPTH=1, CYCLES=20, LFS_BLOCK_CYCLES=1}, + {FILES=3, DEPTH=3, CYCLES=20, LFS_BLOCK_CYCLES=1}, ] code = ''' err = lfs_mount(&lfs, &cfg); From 77e3078b9f28ff689e5a623250eae5c85ae8b3aa Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Wed, 29 Jan 2020 17:50:38 -0600 Subject: [PATCH 17/41] Added/fixed tests for noop writes (where bd error can't be trusted) It's interesting how many ways block devices can show failed writes: 1. prog can error 2. erase can error 3. read can error after writing (ECC failure) 4. prog doesn't error but doesn't write the data correctly 5. erase doesn't error but doesn't erase correctly Can read fail without an error? Yes, though this appears the same as prog and erase failing. These weren't all simulated by testbd since I unintentionally assumed the block device could always error. Fixed by added additional bad-black behaviors to testbd. Note: This also includes a small fix where we can miss bad writes if the underlying block device contains a valid commit with the exact same size in the exact same offset. --- bd/lfs_testbd.c | 31 ++++--- bd/lfs_testbd.h | 20 +++-- lfs.c | 6 +- scripts/test.py | 6 +- tests/test_badblocks.toml | 171 +++++++------------------------------ tests/test_exhaustion.toml | 34 +++----- 6 files changed, 80 insertions(+), 188 deletions(-) diff --git a/bd/lfs_testbd.c b/bd/lfs_testbd.c index 62cba015..8de7dcbc 100644 --- a/bd/lfs_testbd.c +++ b/bd/lfs_testbd.c @@ -47,7 +47,7 @@ int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, memset(bd->wear, 0, sizeof(lfs_testbd_wear_t) * cfg->block_count); } - + // create underlying block device if (bd->persist) { bd->u.file.cfg = (struct lfs_filebd_config){ @@ -155,9 +155,8 @@ int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, LFS_ASSERT(block < cfg->block_count); // block bad? - if (bd->cfg->erase_cycles && - bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_NOREAD && - bd->wear[block] >= bd->cfg->erase_cycles) { + if (bd->cfg->erase_cycles && bd->wear[block] >= bd->cfg->erase_cycles && + bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_READERROR) { LFS_TRACE("lfs_testbd_read -> %d", LFS_ERR_CORRUPT); return LFS_ERR_CORRUPT; } @@ -180,11 +179,18 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, LFS_ASSERT(block < cfg->block_count); // block bad? - if (bd->cfg->erase_cycles && - bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_NOPROG && - bd->wear[block] >= bd->cfg->erase_cycles) { - LFS_TRACE("lfs_testbd_prog -> %d", LFS_ERR_CORRUPT); - return LFS_ERR_CORRUPT; + if (bd->cfg->erase_cycles && bd->wear[block] >= bd->cfg->erase_cycles) { + if (bd->cfg->badblock_behavior == + LFS_TESTBD_BADBLOCK_PROGERROR) { + LFS_TRACE("lfs_testbd_prog -> %d", LFS_ERR_CORRUPT); + return LFS_ERR_CORRUPT; + } else if (bd->cfg->badblock_behavior == + LFS_TESTBD_BADBLOCK_PROGNOOP || + bd->cfg->badblock_behavior == + LFS_TESTBD_BADBLOCK_ERASENOOP) { + LFS_TRACE("lfs_testbd_prog -> %d", 0); + return 0; + } } // prog @@ -219,9 +225,14 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { // block bad? if (bd->cfg->erase_cycles) { if (bd->wear[block] >= bd->cfg->erase_cycles) { - if (bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_NOERASE) { + if (bd->cfg->badblock_behavior == + LFS_TESTBD_BADBLOCK_ERASEERROR) { LFS_TRACE("lfs_testbd_erase -> %d", LFS_ERR_CORRUPT); return LFS_ERR_CORRUPT; + } else if (bd->cfg->badblock_behavior == + LFS_TESTBD_BADBLOCK_ERASENOOP) { + LFS_TRACE("lfs_testbd_erase -> %d", 0); + return 0; } } else { // mark wear diff --git a/bd/lfs_testbd.h b/bd/lfs_testbd.h index 78580d44..68180acc 100644 --- a/bd/lfs_testbd.h +++ b/bd/lfs_testbd.h @@ -19,14 +19,18 @@ extern "C" #endif -// Mode determining how "bad blocks" behave during testing. This -// simulates some real-world circumstances such as writes not -// going through (noprog), erases not sticking (noerase), and ECC -// failures (noread). +// Mode determining how "bad blocks" behave during testing. This simulates +// some real-world circumstances such as progs not sticking (prog-noop), +// a readonly disk (erase-noop), and ECC failures (read-error). +// +// Not that read-noop is not allowed. Read _must_ return a consistent (but +// may be arbitrary) value on every read. enum lfs_testbd_badblock_behavior { - LFS_TESTBD_BADBLOCK_NOPROG = 0, - LFS_TESTBD_BADBLOCK_NOERASE = 1, - LFS_TESTBD_BADBLOCK_NOREAD = 2, + LFS_TESTBD_BADBLOCK_PROGERROR, + LFS_TESTBD_BADBLOCK_ERASEERROR, + LFS_TESTBD_BADBLOCK_READERROR, + LFS_TESTBD_BADBLOCK_PROGNOOP, + LFS_TESTBD_BADBLOCK_ERASENOOP, }; // Type for measuring wear @@ -82,7 +86,7 @@ typedef struct lfs_testbd { /// Block device API /// // Create a test block device using the geometry in lfs_config -// +// // Note that filebd is used if a path is provided, if path is NULL // testbd will use rambd which can be much faster. int lfs_testbd_create(const struct lfs_config *cfg, const char *path); diff --git a/lfs.c b/lfs.c index 0a27e9be..9785092d 100644 --- a/lfs.c +++ b/lfs.c @@ -1300,9 +1300,9 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { // check against written crc to detect if block is readonly // (we may pick up old commits) // TODO rm me? -// if (i == noff && crc != ncrc) { -// return LFS_ERR_CORRUPT; -// } + if (i == noff && crc != ncrc) { + return LFS_ERR_CORRUPT; + } crc = lfs_crc(crc, &dat, 1); } diff --git a/scripts/test.py b/scripts/test.py index 3c3d692d..cbc8838f 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -52,7 +52,7 @@ 'LFS_LOOKAHEAD_SIZE': 16, 'LFS_ERASE_VALUE': 0xff, 'LFS_ERASE_CYCLES': 0, - 'LFS_BADBLOCK_BEHAVIOR': 'LFS_TESTBD_BADBLOCK_NOPROG', + 'LFS_BADBLOCK_BEHAVIOR': 'LFS_TESTBD_BADBLOCK_PROGERROR', } PROLOGUE = """ // prologue @@ -183,12 +183,10 @@ def shouldtest(self, **args): return False elif self.if_ is not None: if_ = self.if_ - print(if_) while True: for k, v in self.defines.items(): if k in if_: if_ = if_.replace(k, '(%s)' % v) - print(if_) break else: break @@ -196,8 +194,6 @@ def shouldtest(self, **args): re.sub('(\&\&|\?)', ' and ', re.sub('(\|\||:)', ' or ', re.sub('!(?!=)', ' not ', if_)))) - print(if_) - print('---', eval(if_), '---') return eval(if_) else: return True diff --git a/tests/test_badblocks.toml b/tests/test_badblocks.toml index 7969d435..44c3e097 100644 --- a/tests/test_badblocks.toml +++ b/tests/test_badblocks.toml @@ -1,72 +1,14 @@ [[case]] # single bad blocks +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_ERASE_CYCLES = 0xffffffff -define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_NOPROG' -define.NAMEMULT = 64 -define.FILEMULT = 1 -code = ''' - for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT; badblock++) { - lfs_testbd_setwear(&cfg, badblock-1, 0) => 0; - lfs_testbd_setwear(&cfg, badblock, 0xffffffff) => 0; - - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - for (int i = 1; i < 10; i++) { - for (int j = 0; j < NAMEMULT; j++) { - buffer[j] = '0'+i; - } - buffer[NAMEMULT] = '\0'; - lfs_mkdir(&lfs, (char*)buffer) => 0; - - buffer[NAMEMULT] = '/'; - for (int j = 0; j < NAMEMULT; j++) { - buffer[j+NAMEMULT+1] = '0'+i; - } - buffer[2*NAMEMULT+1] = '\0'; - lfs_file_open(&lfs, &file, (char*)buffer, - LFS_O_WRONLY | LFS_O_CREAT) => 0; - - size = NAMEMULT; - for (int j = 0; j < i*FILEMULT; j++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; - - lfs_mount(&lfs, &cfg) => 0; - for (int i = 1; i < 10; i++) { - for (int j = 0; j < NAMEMULT; j++) { - buffer[j] = '0'+i; - } - buffer[NAMEMULT] = '\0'; - lfs_stat(&lfs, (char*)buffer, &info) => 0; - info.type => LFS_TYPE_DIR; - - buffer[NAMEMULT] = '/'; - for (int j = 0; j < NAMEMULT; j++) { - buffer[j+NAMEMULT+1] = '0'+i; - } - buffer[2*NAMEMULT+1] = '\0'; - lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; - - size = NAMEMULT; - for (int j = 0; j < i*FILEMULT; j++) { - uint8_t rbuffer[1024]; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(buffer, rbuffer, size) => 0; - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; - } -''' - -[[case]] # single persistent blocks (can't erase) -define.LFS_ERASE_CYCLES = 0xffffffff -define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_NOERASE' +define.LFS_ERASE_VALUE = [0x00, 0xff, -1] +define.LFS_BADBLOCK_BEHAVIOR = [ + 'LFS_TESTBD_BADBLOCK_PROGERROR', + 'LFS_TESTBD_BADBLOCK_ERASEERROR', + 'LFS_TESTBD_BADBLOCK_READERROR', + 'LFS_TESTBD_BADBLOCK_PROGNOOP', + 'LFS_TESTBD_BADBLOCK_ERASENOOP', +] define.NAMEMULT = 64 define.FILEMULT = 1 code = ''' @@ -130,78 +72,16 @@ code = ''' } ''' -[[case]] # single unreadable blocks (can't read) -define.LFS_ERASE_CYCLES = 0xffffffff -define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_NOREAD' -define.NAMEMULT = 64 -define.FILEMULT = 1 -code = ''' - for (lfs_block_t badblock = 2; badblock < LFS_BLOCK_COUNT/2; badblock++) { - lfs_testbd_setwear(&cfg, badblock-1, 0) => 0; - lfs_testbd_setwear(&cfg, badblock, 0xffffffff) => 0; - - lfs_format(&lfs, &cfg) => 0; - - lfs_mount(&lfs, &cfg) => 0; - for (int i = 1; i < 10; i++) { - for (int j = 0; j < NAMEMULT; j++) { - buffer[j] = '0'+i; - } - buffer[NAMEMULT] = '\0'; - lfs_mkdir(&lfs, (char*)buffer) => 0; - - buffer[NAMEMULT] = '/'; - for (int j = 0; j < NAMEMULT; j++) { - buffer[j+NAMEMULT+1] = '0'+i; - } - buffer[2*NAMEMULT+1] = '\0'; - lfs_file_open(&lfs, &file, (char*)buffer, - LFS_O_WRONLY | LFS_O_CREAT) => 0; - - size = NAMEMULT; - for (int j = 0; j < i*FILEMULT; j++) { - lfs_file_write(&lfs, &file, buffer, size) => size; - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; - - lfs_mount(&lfs, &cfg) => 0; - for (int i = 1; i < 10; i++) { - for (int j = 0; j < NAMEMULT; j++) { - buffer[j] = '0'+i; - } - buffer[NAMEMULT] = '\0'; - lfs_stat(&lfs, (char*)buffer, &info) => 0; - info.type => LFS_TYPE_DIR; - - buffer[NAMEMULT] = '/'; - for (int j = 0; j < NAMEMULT; j++) { - buffer[j+NAMEMULT+1] = '0'+i; - } - buffer[2*NAMEMULT+1] = '\0'; - lfs_file_open(&lfs, &file, (char*)buffer, LFS_O_RDONLY) => 0; - - size = NAMEMULT; - for (int j = 0; j < i*FILEMULT; j++) { - uint8_t rbuffer[1024]; - lfs_file_read(&lfs, &file, rbuffer, size) => size; - memcmp(buffer, rbuffer, size) => 0; - } - - lfs_file_close(&lfs, &file) => 0; - } - lfs_unmount(&lfs) => 0; - } -''' - [[case]] # region corruption (causes cascading failures) +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_ERASE_VALUE = [0x00, 0xff, -1] define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', + 'LFS_TESTBD_BADBLOCK_PROGERROR', + 'LFS_TESTBD_BADBLOCK_ERASEERROR', + 'LFS_TESTBD_BADBLOCK_READERROR', + 'LFS_TESTBD_BADBLOCK_PROGNOOP', + 'LFS_TESTBD_BADBLOCK_ERASENOOP', ] define.NAMEMULT = 64 define.FILEMULT = 1 @@ -266,11 +146,15 @@ code = ''' ''' [[case]] # alternating corruption (causes cascading failures) +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_ERASE_VALUE = [0x00, 0xff, -1] define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', + 'LFS_TESTBD_BADBLOCK_PROGERROR', + 'LFS_TESTBD_BADBLOCK_ERASEERROR', + 'LFS_TESTBD_BADBLOCK_READERROR', + 'LFS_TESTBD_BADBLOCK_PROGNOOP', + 'LFS_TESTBD_BADBLOCK_ERASENOOP', ] define.NAMEMULT = 64 define.FILEMULT = 1 @@ -337,10 +221,13 @@ code = ''' # other corner cases [[case]] # bad superblocks (corrupt 1 or 0) define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_ERASE_VALUE = [0x00, 0xff, -1] define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', + 'LFS_TESTBD_BADBLOCK_PROGERROR', + 'LFS_TESTBD_BADBLOCK_ERASEERROR', + 'LFS_TESTBD_BADBLOCK_READERROR', + 'LFS_TESTBD_BADBLOCK_PROGNOOP', + 'LFS_TESTBD_BADBLOCK_ERASENOOP', ] code = ''' lfs_testbd_setwear(&cfg, 0, 0xffffffff) => 0; diff --git a/tests/test_exhaustion.toml b/tests/test_exhaustion.toml index 7faa1764..86bf15cb 100644 --- a/tests/test_exhaustion.toml +++ b/tests/test_exhaustion.toml @@ -1,11 +1,13 @@ [[case]] # test running a filesystem to exhaustion define.LFS_ERASE_CYCLES = 10 -define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', + 'LFS_TESTBD_BADBLOCK_PROGERROR', + 'LFS_TESTBD_BADBLOCK_ERASEERROR', + 'LFS_TESTBD_BADBLOCK_READERROR', + 'LFS_TESTBD_BADBLOCK_PROGNOOP', + 'LFS_TESTBD_BADBLOCK_ERASENOOP', ] define.FILES = 10 code = ''' @@ -79,12 +81,14 @@ exhausted: [[case]] # test running a filesystem to exhaustion # which also requires expanding superblocks define.LFS_ERASE_CYCLES = 10 -define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', + 'LFS_TESTBD_BADBLOCK_PROGERROR', + 'LFS_TESTBD_BADBLOCK_ERASEERROR', + 'LFS_TESTBD_BADBLOCK_READERROR', + 'LFS_TESTBD_BADBLOCK_PROGNOOP', + 'LFS_TESTBD_BADBLOCK_ERASENOOP', ] define.FILES = 10 code = ''' @@ -159,13 +163,8 @@ exhausted: [[case]] # wear-level test running a filesystem to exhaustion define.LFS_ERASE_CYCLES = 20 -define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' -define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', -] define.FILES = 10 code = ''' uint32_t run_cycles[2]; @@ -252,13 +251,8 @@ exhausted: [[case]] # wear-level test + expanding superblock define.LFS_ERASE_CYCLES = 20 -define.LFS_BLOCK_COUNT = 256 # small bd so it runs faster +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_BLOCK_CYCLES = 'LFS_ERASE_CYCLES / 2' -define.LFS_BADBLOCK_BEHAVIOR = [ - 'LFS_TESTBD_BADBLOCK_NOPROG', - 'LFS_TESTBD_BADBLOCK_NOERASE', - 'LFS_TESTBD_BADBLOCK_NOREAD', -] define.FILES = 10 code = ''' uint32_t run_cycles[2]; From 44d7112794d49bb2c56d4ed85b154770cf208c49 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Wed, 29 Jan 2020 17:56:58 -0600 Subject: [PATCH 18/41] Fixed tests/*.toml.* in .gitignore Running test.py creates a log of garbage here --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 1e12cca5..9cafad9c 100644 --- a/.gitignore +++ b/.gitignore @@ -7,4 +7,4 @@ blocks/ lfs test.c -tests_/*.toml.* +tests/*.toml.* From f9c2fd93f289a3d8f23ffebc6f4089260c417258 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Wed, 29 Jan 2020 22:05:58 -0600 Subject: [PATCH 19/41] Removed file outlining on ENOSPC in lfs_file_sync This was initially added as protection against the case where a file grew to no longer fit in a metadata-pair. While in most cases this should be caught by the math in lfs_file_write, it doesn't handle a problem that can happen if the files metadata is large enough that even small inline files can't fit. This can happen if you combine a small block size with large file names and many custom attributes. But trying to outline on ENOSPC creates creates a lot of problems. If we are actually low on space, this is one of the worst things we can do. Inline files take up less space than CTZ skip-lists, but inline files are rendered useless if we outline inline files as soon as we run low on space. On top of this, the outlining logic tries multiple mdir commits if it gets ENOSPC, which can hide errors if ENOSPC is returned for other reasons. In a perfect world, we would be using a different error code for no-room-in-metadata-pair, and no-blocks-on-disk. For now I've removed the outlining logic and we will need to figure out how to handle this situation more robustly. --- lfs.c | 47 +++++++++++++++++++++++------------------------ 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/lfs.c b/lfs.c index 9785092d..be6612f8 100644 --- a/lfs.c +++ b/lfs.c @@ -1631,33 +1631,17 @@ static int lfs_dir_compact(lfs_t *lfs, // TODO huh? LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0); - // update gstate - lfs->gdelta = (lfs_gstate_t){0}; - if (!relocated) { - lfs->gdisk = lfs->gstate; - } - - // TODO here?? - if (relocated) { - // update references if we relocated - LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, - oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - err = lfs_fs_relocate(lfs, oldpair, dir->pair); - if (err) { - // TODO make better - dir->pair[1] = oldpair[1]; // - return err; - } - LFS_DEBUG("Relocated %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, - oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - } - // successful compaction, swap dir pair to indicate most recent lfs_pair_swap(dir->pair); dir->rev = nrev; dir->count = end - begin; dir->off = commit.off; dir->etag = commit.ptag; + // update gstate + lfs->gdelta = (lfs_gstate_t){0}; + if (!relocated) { + lfs->gdisk = lfs->gstate; + } } break; @@ -1685,6 +1669,21 @@ static int lfs_dir_compact(lfs_t *lfs, continue; } + // TODO here?? + if (relocated) { + // update references if we relocated + LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + int err = lfs_fs_relocate(lfs, oldpair, dir->pair); + if (err) { + // TODO make better + //dir->pair[1] = oldpair[1]; // + return err; + } + LFS_DEBUG("Relocated %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + } + return 0; } @@ -2739,9 +2738,9 @@ int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) { {LFS_MKTAG(LFS_FROM_USERATTRS, file->id, file->cfg->attr_count), file->cfg->attrs})); if (err) { - if (err == LFS_ERR_NOSPC && (file->flags & LFS_F_INLINE)) { - goto relocate; - } +// if (err == LFS_ERR_NOSPC && (file->flags & LFS_F_INLINE)) { +// goto relocate; +// } file->flags |= LFS_F_ERRED; LFS_TRACE("lfs_file_sync -> %d", err); return err; From 6a550844f448f608c835d45df4656b972854a130 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Thu, 30 Jan 2020 16:05:42 -0600 Subject: [PATCH 20/41] Modified readmdir/readtree to make reading non-truncated data easier Added indention so there was a more clear separation between the tag description and tag data. Also took the best parts of readmdir.py and added it to readtree.py. Initially I was thinking it was best for these to have completely independent data representations, since you could always call readtree to get more info, but this becomes tedius when needed to look at low-level tag info across multiple directories on the filesystem. --- scripts/readmdir.py | 37 ++++++++++++++------------- scripts/readtree.py | 61 ++++++++++----------------------------------- 2 files changed, 33 insertions(+), 65 deletions(-) diff --git a/scripts/readmdir.py b/scripts/readmdir.py index 1730f029..ef634a96 100755 --- a/scripts/readmdir.py +++ b/scripts/readmdir.py @@ -2,6 +2,7 @@ import struct import binascii +import sys import itertools as it TAG_TYPES = { @@ -271,37 +272,39 @@ def __getitem__(self, args): raise KeyError(gmask, gtag) - def _dump_tags(self, tags, truncate=True): - sys.stdout.write("%-8s %-8s %-13s %4s %4s %s\n" % ( - 'off', 'tag', 'type', 'id', 'len', - 'data (truncated)' if truncate else 12*' '+'data')) + def _dump_tags(self, tags, f=sys.stdout, truncate=True): + f.write("%-8s %-8s %-13s %4s %4s" % ( + 'off', 'tag', 'type', 'id', 'len')) + if truncate: + f.write(' data (truncated)') + f.write('\n') for tag in tags: - sys.stdout.write("%08x: %08x %-13s %4s %4s" % ( + f.write("%08x: %08x %-13s %4s %4s" % ( tag.off, tag, tag.typerepr(), tag.idrepr(), tag.sizerepr())) if truncate: - sys.stdout.write(" %-23s %-8s\n" % ( + f.write(" %-23s %-8s\n" % ( ' '.join('%02x' % c for c in tag.data[:8]), ''.join(c if c >= ' ' and c <= '~' else '.' for c in map(chr, tag.data[:8])))) else: - sys.stdout.write("\n") + f.write("\n") for i in range(0, len(tag.data), 16): - sys.stdout.write("%08x: %-47s %-16s\n" % ( + f.write(" %08x: %-47s %-16s\n" % ( tag.off+i, ' '.join('%02x' % c for c in tag.data[i:i+16]), ''.join(c if c >= ' ' and c <= '~' else '.' for c in map(chr, tag.data[i:i+16])))) - def dump_tags(self, truncate=True): - self._dump_tags(self.tags, truncate=truncate) + def dump_tags(self, f=sys.stdout, truncate=True): + self._dump_tags(self.tags, f=f, truncate=truncate) - def dump_log(self, truncate=True): - self._dump_tags(self.log, truncate=truncate) + def dump_log(self, f=sys.stdout, truncate=True): + self._dump_tags(self.log, f=f, truncate=truncate) - def dump_all(self, truncate=True): - self._dump_tags(self.all_, truncate=truncate) + def dump_all(self, f=sys.stdout, truncate=True): + self._dump_tags(self.all_, f=f, truncate=truncate) def main(args): blocks = [] @@ -337,10 +340,10 @@ def main(args): help="First block address for finding the metadata pair.") parser.add_argument('block2', nargs='?', type=lambda x: int(x, 0), help="Second block address for finding the metadata pair.") - parser.add_argument('-a', '--all', action='store_true', - help="Show all tags in log, included tags in corrupted commits.") parser.add_argument('-l', '--log', action='store_true', help="Show tags in log.") + parser.add_argument('-a', '--all', action='store_true', + help="Show all tags in log, included tags in corrupted commits.") parser.add_argument('-T', '--no-truncate', action='store_true', - help="Don't truncate large amounts of data in tags.") + help="Don't truncate large amounts of data.") sys.exit(main(parser.parse_args())) diff --git a/scripts/readtree.py b/scripts/readtree.py index ecfdab94..ea8cb5f5 100755 --- a/scripts/readtree.py +++ b/scripts/readtree.py @@ -13,45 +13,6 @@ def popc(x): def ctz(x): return len(bin(x)) - len(bin(x).rstrip('0')) -def dumptags(args, mdir, f): - if args.all: - tags = mdir.all_ - elif args.log: - tags = mdir.log - else: - tags = mdir.tags - - for k, tag in enumerate(tags): - f.write("tag %08x %s" % (tag, tag.typerepr())) - if tag.id != 0x3ff: - f.write(" id %d" % tag.id) - if tag.size != 0x3ff: - f.write(" size %d" % tag.size) - if tag.is_('name'): - f.write(" name %s" % - json.dumps(tag.data.decode('utf8'))) - if tag.is_('dirstruct'): - f.write(" dir {%#x, %#x}" % struct.unpack( - '= ' ' and c <= '~' else '.' - for c in map(chr, tag.data[i:i+16])))) - def dumpentries(args, mdir, f): for k, id_ in enumerate(mdir.ids): name = mdir[Tag('name', id_, 0)] @@ -72,8 +33,8 @@ def dumpentries(args, mdir, f): if args.data and struct_.is_('inlinestruct'): for i in range(0, len(struct_.data), 16): - f.write(" %-47s %-16s\n" % ( - ' '.join('%02x' % c for c in struct_.data[i:i+16]), + f.write(" %08x: %-47s %-16s\n" % ( + i, ' '.join('%02x' % c for c in struct_.data[i:i+16]), ''.join(c if c >= ' ' and c <= '~' else '.' for c in map(chr, struct_.data[i:i+16])))) elif args.data and struct_.is_('ctzstruct'): @@ -95,8 +56,8 @@ def dumpentries(args, mdir, f): it.chain.from_iterable(reversed(data)), size)) for i in range(0, min(len(data), 256) if not args.no_truncate else len(data), 16): - f.write(" %-47s %-16s\n" % ( - ' '.join('%02x' % c for c in data[i:i+16]), + f.write(" %08x: %-47s %-16s\n" % ( + i, ' '.join('%02x' % c for c in data[i:i+16]), ''.join(c if c >= ' ' and c <= '~' else '.' for c in map(chr, data[i:i+16])))) @@ -239,8 +200,12 @@ def main(args): ' (corrupted)' if not mdir else '')) f = io.StringIO() - if args.tags or args.all or args.log: - dumptags(args, mdir, f) + if args.tags: + mdir.dump_tags(f, truncate=not args.no_truncate) + elif args.log: + mdir.dump_log(f, truncate=not args.no_truncate) + elif args.all: + mdir.dump_all(f, truncate=not args.no_truncate) else: dumpentries(args, mdir, f) @@ -285,12 +250,12 @@ def main(args): help="Show contents of metadata-pairs/directories.") parser.add_argument('-t', '--tags', action='store_true', help="Show metadata tags instead of reconstructing entries.") - parser.add_argument('-a', '--all', action='store_true', - help="Show all tags in log, included tags in corrupted commits.") parser.add_argument('-l', '--log', action='store_true', help="Show tags in log.") + parser.add_argument('-a', '--all', action='store_true', + help="Show all tags in log, included tags in corrupted commits.") parser.add_argument('-d', '--data', action='store_true', help="Also show the raw contents of files/attrs/tags.") parser.add_argument('-T', '--no-truncate', action='store_true', - help="Don't truncate large amounts of data in files.") + help="Don't truncate large amounts of data.") sys.exit(main(parser.parse_args())) From fe957de892edbefd7af54acbe19e9a4e0f0fb1d0 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 9 Feb 2020 08:52:20 -0600 Subject: [PATCH 21/41] Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable. --- lfs.c | 52 ++++++++------ tests/test_exhaustion.toml | 134 +++++++++++++++++++++++++++++++++---- 2 files changed, 153 insertions(+), 33 deletions(-) diff --git a/lfs.c b/lfs.c index be6612f8..74fafd9e 100644 --- a/lfs.c +++ b/lfs.c @@ -1486,7 +1486,7 @@ static int lfs_dir_compact(lfs_t *lfs, // increment revision count uint32_t nrev = dir->rev + 1; if (lfs->cfg->block_cycles > 0 && - (nrev % (lfs->cfg->block_cycles+1) == 0)) { + (nrev % ((lfs->cfg->block_cycles+1)|1) == 0)) { if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) { // oh no! we're writing too much to the superblock, // should we expand? @@ -1524,6 +1524,7 @@ static int lfs_dir_compact(lfs_t *lfs, } else { // we're writing too much, time to relocate tired = true; + //nrev += 1;//lfs->cfg->block_cycles & 1; // TODO do this here? goto relocate; } } @@ -1631,17 +1632,33 @@ static int lfs_dir_compact(lfs_t *lfs, // TODO huh? LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0); + // update gstate + lfs->gdelta = (lfs_gstate_t){0}; + if (!relocated) { + lfs->gdisk = lfs->gstate; + } + + // TODO here?? + if (relocated) { + // update references if we relocated + LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + int err = lfs_fs_relocate(lfs, oldpair, dir->pair); + if (err) { + // TODO make better + //dir->pair[1] = oldpair[1]; // + return err; + } +// LFS_DEBUG("Relocated %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, +// oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + } + // successful compaction, swap dir pair to indicate most recent lfs_pair_swap(dir->pair); dir->rev = nrev; dir->count = end - begin; dir->off = commit.off; dir->etag = commit.ptag; - // update gstate - lfs->gdelta = (lfs_gstate_t){0}; - if (!relocated) { - lfs->gdisk = lfs->gstate; - } } break; @@ -1669,20 +1686,6 @@ static int lfs_dir_compact(lfs_t *lfs, continue; } - // TODO here?? - if (relocated) { - // update references if we relocated - LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, - oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - int err = lfs_fs_relocate(lfs, oldpair, dir->pair); - if (err) { - // TODO make better - //dir->pair[1] = oldpair[1]; // - return err; - } - LFS_DEBUG("Relocated %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, - oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - } return 0; } @@ -1709,6 +1712,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, // calculate changes to the directory bool hasdelete = false; + lfs_mdir_t olddir = *dir; // TODO is this a good idea? for (int i = 0; i < attrcount; i++) { if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE) { dir->count += 1; @@ -1729,6 +1733,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, lfs_mdir_t pdir; int err = lfs_fs_pred(lfs, dir->pair, &pdir); if (err && err != LFS_ERR_NOENT) { + *dir = olddir; return err; } @@ -1761,6 +1766,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) { goto compact; } + *dir = olddir; return err; } @@ -1773,6 +1779,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, if (!lfs_gstate_iszero(&delta)) { err = lfs_dir_getgstate(lfs, dir, &delta); if (err) { + *dir = olddir; return err; } @@ -1784,6 +1791,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) { goto compact; } + *dir = olddir; return err; } } @@ -1794,6 +1802,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) { goto compact; } + *dir = olddir; return err; } @@ -1812,6 +1821,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, int err = lfs_dir_compact(lfs, dir, attrs, attrcount, dir, 0, dir->count); if (err) { + *dir = olddir; return err; } } @@ -3845,7 +3855,7 @@ int lfs_fs_traverse(lfs_t *lfs, LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } - } else if (lfs_gstate_hasorphans(&lfs->gstate) && + } else if (/*lfs_gstate_hasorphans(&lfs->gstate) && TODO maybe report size-dirs/2 ? */ lfs_tag_type3(tag) == LFS_TYPE_DIRSTRUCT) { // TODO HMMMMMM HMMMMMMMMMMMMMMMMMMM for (int i = 0; i < 2; i++) { diff --git a/tests/test_exhaustion.toml b/tests/test_exhaustion.toml index 86bf15cb..0460071e 100644 --- a/tests/test_exhaustion.toml +++ b/tests/test_exhaustion.toml @@ -42,7 +42,7 @@ code = ''' if (err == LFS_ERR_NOSPC) { goto exhausted; } - } + } for (uint32_t i = 0; i < FILES; i++) { // check for errors @@ -59,7 +59,7 @@ code = ''' } lfs_file_close(&lfs, &file) => 0; - } + } lfs_unmount(&lfs) => 0; cycle += 1; @@ -72,7 +72,7 @@ exhausted: // check for errors sprintf(path, "roadrunner/test%d", i); lfs_stat(&lfs, path, &info) => 0; - } + } lfs_unmount(&lfs) => 0; LFS_WARN("completed %d cycles", cycle); @@ -120,7 +120,7 @@ code = ''' if (err == LFS_ERR_NOSPC) { goto exhausted; } - } + } for (uint32_t i = 0; i < FILES; i++) { // check for errors @@ -137,7 +137,7 @@ code = ''' } lfs_file_close(&lfs, &file) => 0; - } + } lfs_unmount(&lfs) => 0; cycle += 1; @@ -150,7 +150,7 @@ exhausted: // check for errors sprintf(path, "test%d", i); lfs_stat(&lfs, path, &info) => 0; - } + } lfs_unmount(&lfs) => 0; LFS_WARN("completed %d cycles", cycle); @@ -207,7 +207,7 @@ code = ''' if (err == LFS_ERR_NOSPC) { goto exhausted; } - } + } for (uint32_t i = 0; i < FILES; i++) { // check for errors @@ -224,7 +224,7 @@ code = ''' } lfs_file_close(&lfs, &file) => 0; - } + } lfs_unmount(&lfs) => 0; cycle += 1; @@ -237,7 +237,7 @@ exhausted: // check for errors sprintf(path, "roadrunner/test%d", i); lfs_stat(&lfs, path, &info) => 0; - } + } lfs_unmount(&lfs) => 0; run_cycles[run] = cycle; @@ -292,7 +292,7 @@ code = ''' if (err == LFS_ERR_NOSPC) { goto exhausted; } - } + } for (uint32_t i = 0; i < FILES; i++) { // check for errors @@ -309,7 +309,7 @@ code = ''' } lfs_file_close(&lfs, &file) => 0; - } + } lfs_unmount(&lfs) => 0; cycle += 1; @@ -322,7 +322,7 @@ exhausted: // check for errors sprintf(path, "test%d", i); lfs_stat(&lfs, path, &info) => 0; - } + } lfs_unmount(&lfs) => 0; run_cycles[run] = cycle; @@ -333,3 +333,113 @@ exhausted: // check we increased the lifetime by 2x with ~10% error LFS_ASSERT(run_cycles[1]*110/100 > 2*run_cycles[0]); ''' + +[[case]] # test that we wear blocks roughly evenly +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster +define.LFS_BLOCK_CYCLES = [5, 4, 3, 2, 1] +#define.LFS_BLOCK_CYCLES = [4, 2] +define.CYCLES = 100 +define.FILES = 10 +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "roadrunner") => 0; + lfs_unmount(&lfs) => 0; + + uint32_t cycle = 0; + while (cycle < CYCLES) { + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // chose name, roughly random seed, and random 2^n size + sprintf(path, "roadrunner/test%d", i); + srand(cycle * i); + size = 1 << 4; //((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC) => 0; + + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); + assert(res == 1 || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + if (err == LFS_ERR_NOSPC) { + goto exhausted; + } + } + + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "roadrunner/test%d", i); + srand(cycle * i); + size = 1 << 4; //((rand() % 10)+2); + + lfs_file_open(&lfs, &file, path, LFS_O_RDONLY) => 0; + for (lfs_size_t j = 0; j < size; j++) { + char c = 'a' + (rand() % 26); + char r; + lfs_file_read(&lfs, &file, &r, 1) => 1; + assert(r == c); + } + + lfs_file_close(&lfs, &file) => 0; + } + lfs_unmount(&lfs) => 0; + + cycle += 1; + } + +exhausted: + // should still be readable + lfs_mount(&lfs, &cfg) => 0; + for (uint32_t i = 0; i < FILES; i++) { + // check for errors + sprintf(path, "roadrunner/test%d", i); + lfs_stat(&lfs, path, &info) => 0; + } + lfs_unmount(&lfs) => 0; + + LFS_WARN("completed %d cycles", cycle); + + // check the wear on our block device + lfs_testbd_wear_t minwear = -1; + lfs_testbd_wear_t totalwear = 0; + lfs_testbd_wear_t maxwear = 0; + // skip 0 and 1 as superblock movement is intentionally avoided + for (lfs_block_t b = 2; b < LFS_BLOCK_COUNT; b++) { + lfs_testbd_wear_t wear = lfs_testbd_getwear(&cfg, b); + printf("%08x: wear %d\n", b, wear); + assert(wear >= 0); + if (wear < minwear) { + minwear = wear; + } + if (wear > maxwear) { + maxwear = wear; + } + totalwear += wear; + } + lfs_testbd_wear_t avgwear = totalwear / LFS_BLOCK_COUNT; + LFS_WARN("max wear: %d cycles", maxwear); + LFS_WARN("avg wear: %d cycles", totalwear / LFS_BLOCK_COUNT); + LFS_WARN("min wear: %d cycles", minwear); + + // find standard deviation^2 + lfs_testbd_wear_t dev2 = 0; + for (lfs_block_t b = 2; b < LFS_BLOCK_COUNT; b++) { + lfs_testbd_wear_t wear = lfs_testbd_getwear(&cfg, b); + assert(wear >= 0); + lfs_testbd_swear_t diff = wear - avgwear; + dev2 += diff*diff; + } + dev2 /= totalwear; + LFS_WARN("std dev^2: %d", dev2); + assert(dev2 < 8); +''' + From 6530cb3a6129f28dda117fdcb0231291b0eff680 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 9 Feb 2020 09:05:37 -0600 Subject: [PATCH 22/41] Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem. --- lfs.c | 36 ++++++++++++++++++------------------ tests/test_orphans.toml | 4 ++-- 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/lfs.c b/lfs.c index 74fafd9e..2f4593ae 100644 --- a/lfs.c +++ b/lfs.c @@ -414,6 +414,9 @@ static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t dir[2], lfs_mdir_t *parent); static int lfs_fs_relocate(lfs_t *lfs, const lfs_block_t oldpair[2], lfs_block_t newpair[2]); +int lfs_fs_traverseraw(lfs_t *lfs, + int (*cb)(void *data, lfs_block_t block), void *data, + bool includeorphans); static int lfs_fs_forceconsistency(lfs_t *lfs); static int lfs_deinit(lfs_t *lfs); #ifdef LFS_MIGRATE @@ -472,7 +475,7 @@ static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) { // find mask of free blocks from tree memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size); - int err = lfs_fs_traverse(lfs, lfs_alloc_lookahead, lfs); + int err = lfs_fs_traverseraw(lfs, lfs_alloc_lookahead, lfs, true); if (err) { return err; } @@ -3798,10 +3801,9 @@ int lfs_unmount(lfs_t *lfs) { /// Filesystem filesystem operations /// -int lfs_fs_traverse(lfs_t *lfs, - int (*cb)(void *data, lfs_block_t block), void *data) { - LFS_TRACE("lfs_fs_traverse(%p, %p, %p)", - (void*)lfs, (void*)(uintptr_t)cb, data); +int lfs_fs_traverseraw(lfs_t *lfs, + int (*cb)(void *data, lfs_block_t block), void *data, + bool includeorphans) { // iterate over metadata pairs lfs_mdir_t dir = {.tail = {0, 1}}; @@ -3810,7 +3812,6 @@ int lfs_fs_traverse(lfs_t *lfs, if (lfs->lfs1) { int err = lfs1_traverse(lfs, cb, data); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } @@ -3823,7 +3824,6 @@ int lfs_fs_traverse(lfs_t *lfs, for (int i = 0; i < 2; i++) { int err = cb(data, dir.tail[i]); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } } @@ -3831,7 +3831,6 @@ int lfs_fs_traverse(lfs_t *lfs, // iterate through ids in directory int err = lfs_dir_fetch(lfs, &dir, dir.tail); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } @@ -3843,7 +3842,6 @@ int lfs_fs_traverse(lfs_t *lfs, if (tag == LFS_ERR_NOENT) { continue; } - LFS_TRACE("lfs_fs_traverse -> %"PRId32, tag); return tag; } lfs_ctz_fromle32(&ctz); @@ -3852,17 +3850,13 @@ int lfs_fs_traverse(lfs_t *lfs, err = lfs_ctz_traverse(lfs, NULL, &lfs->rcache, ctz.head, ctz.size, cb, data); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } - } else if (/*lfs_gstate_hasorphans(&lfs->gstate) && TODO maybe report size-dirs/2 ? */ + } else if (includeorphans && lfs_tag_type3(tag) == LFS_TYPE_DIRSTRUCT) { - // TODO HMMMMMM HMMMMMMMMMMMMMMMMMMM for (int i = 0; i < 2; i++) { - //printf("HMM %x\n", (&ctz.head)[i]); err = cb(data, (&ctz.head)[i]); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } } @@ -3880,7 +3874,6 @@ int lfs_fs_traverse(lfs_t *lfs, int err = lfs_ctz_traverse(lfs, &f->cache, &lfs->rcache, f->ctz.head, f->ctz.size, cb, data); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } } @@ -3889,16 +3882,23 @@ int lfs_fs_traverse(lfs_t *lfs, int err = lfs_ctz_traverse(lfs, &f->cache, &lfs->rcache, f->block, f->pos, cb, data); if (err) { - LFS_TRACE("lfs_fs_traverse -> %d", err); return err; } } } - LFS_TRACE("lfs_fs_traverse -> %d", 0); return 0; } +int lfs_fs_traverse(lfs_t *lfs, + int (*cb)(void *data, lfs_block_t block), void *data) { + LFS_TRACE("lfs_fs_traverse(%p, %p, %p)", + (void*)lfs, (void*)(uintptr_t)cb, data); + int err = lfs_fs_traverseraw(lfs, cb, data, true); + LFS_TRACE("lfs_fs_traverse -> %d", 0); + return err; +} + static int lfs_fs_pred(lfs_t *lfs, const lfs_block_t pair[2], lfs_mdir_t *pdir) { // iterate over all directory directory entries @@ -4207,7 +4207,7 @@ static int lfs_fs_size_count(void *p, lfs_block_t block) { lfs_ssize_t lfs_fs_size(lfs_t *lfs) { LFS_TRACE("lfs_fs_size(%p)", (void*)lfs); lfs_size_t size = 0; - int err = lfs_fs_traverse(lfs, lfs_fs_size_count, &size); + int err = lfs_fs_traverseraw(lfs, lfs_fs_size_count, &size, false); if (err) { LFS_TRACE("lfs_fs_size -> %d", err); return err; diff --git a/tests/test_orphans.toml b/tests/test_orphans.toml index ea4ca507..8d0f3deb 100644 --- a/tests/test_orphans.toml +++ b/tests/test_orphans.toml @@ -31,13 +31,13 @@ code = ''' lfs_mount(&lfs, &cfg) => 0; lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; lfs_stat(&lfs, "parent/child", &info) => 0; - lfs_fs_size(&lfs) => 12; + lfs_fs_size(&lfs) => 8; lfs_unmount(&lfs) => 0; lfs_mount(&lfs, &cfg) => 0; lfs_stat(&lfs, "parent/orphan", &info) => LFS_ERR_NOENT; lfs_stat(&lfs, "parent/child", &info) => 0; - lfs_fs_size(&lfs) => 12; + lfs_fs_size(&lfs) => 8; // this mkdir should both create a dir and deorphan, so size // should be unchanged lfs_mkdir(&lfs, "parent/otherchild") => 0; From 02c84ac5f4da1a2fdb7aa15cc11afcddca2f3ea8 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 9 Feb 2020 10:02:41 -0600 Subject: [PATCH 23/41] Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512 --- lfs.c | 177 ++++++++++++++++--------------------- tests/test_dirs.toml | 2 +- tests/test_exhaustion.toml | 1 - 3 files changed, 77 insertions(+), 103 deletions(-) diff --git a/lfs.c b/lfs.c index 2f4593ae..5ad81724 100644 --- a/lfs.c +++ b/lfs.c @@ -805,13 +805,12 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, tag = lfs_frombe32(tag) ^ ptag; // next commit not yet programmed or we're not in valid range - if (!lfs_tag_isvalid(tag) || - off + lfs_tag_dsize(tag) > lfs->cfg->block_size) { - //printf("read block %d valid %d ntag %08x ptag %08x off %d (off %d dsize %dblock %d)\n", dir->pair[0], lfs_tag_isvalid(tag), tag, ptag, dir->off, off, lfs_tag_dsize(tag), lfs->cfg->block_size); - //printf("read block %d erased = %d\n", dir->pair[0], (lfs_tag_type1(ptag) == LFS_TYPE_CRC && dir->off % lfs->cfg->prog_size == 0)); + if (!lfs_tag_isvalid(tag)) { dir->erased = (lfs_tag_type1(ptag) == LFS_TYPE_CRC && - dir->off % lfs->cfg->prog_size == 0 && - !lfs_tag_isvalid(tag)); + dir->off % lfs->cfg->prog_size == 0); + break; + } else if (off + lfs_tag_dsize(tag) > lfs->cfg->block_size) { + dir->erased = false; break; } @@ -1302,7 +1301,6 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { // check against written crc to detect if block is readonly // (we may pick up old commits) -// TODO rm me? if (i == noff && crc != ncrc) { return LFS_ERR_CORRUPT; } @@ -1403,7 +1401,6 @@ static int lfs_dir_split(lfs_t *lfs, return err; } - //dir->rev += 1; // TODO really? dir->tail[0] = tail.pair[0]; dir->tail[1] = tail.pair[1]; dir->split = true; @@ -1487,9 +1484,15 @@ static int lfs_dir_compact(lfs_t *lfs, } // increment revision count - uint32_t nrev = dir->rev + 1; + dir->rev += 1; + // If our revision count == n * block_cycles, we should force a relocation, + // this is how littlefs wear-levels at the metadata-pair level. Note that we + // actually use (block_cycles+1)|1, this is to avoid two corner cases: + // 1. block_cycles = 1, which would prevent relocations from terminating + // 2. block_cycles = 2n, which, due to aliasing, would only ever relocate + // one metadata block in the pair, effectively making this useless if (lfs->cfg->block_cycles > 0 && - (nrev % ((lfs->cfg->block_cycles+1)|1) == 0)) { + (dir->rev % ((lfs->cfg->block_cycles+1)|1) == 0)) { if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) { // oh no! we're writing too much to the superblock, // should we expand? @@ -1501,8 +1504,7 @@ static int lfs_dir_compact(lfs_t *lfs, // do we have extra space? littlefs can't reclaim this space // by itself, so expand cautiously if ((lfs_size_t)res < lfs->cfg->block_count/2) { - LFS_DEBUG("Expanding superblock at rev %"PRIu32, nrev); - //dir->rev += 1; // TODO hmm + LFS_DEBUG("Expanding superblock at rev %"PRIu32, dir->rev); int err = lfs_dir_split(lfs, dir, attrs, attrcount, source, begin, end); if (err && err != LFS_ERR_NOSPC) { @@ -1527,7 +1529,6 @@ static int lfs_dir_compact(lfs_t *lfs, } else { // we're writing too much, time to relocate tired = true; - //nrev += 1;//lfs->cfg->block_cycles & 1; // TODO do this here? goto relocate; } } @@ -1556,10 +1557,10 @@ static int lfs_dir_compact(lfs_t *lfs, } // write out header - nrev = lfs_tole32(nrev); + dir->rev = lfs_tole32(dir->rev); err = lfs_dir_commitprog(lfs, &commit, - &nrev, sizeof(nrev)); - nrev = lfs_fromle32(nrev); + &dir->rev, sizeof(dir->rev)); + dir->rev = lfs_fromle32(dir->rev); if (err) { if (err == LFS_ERR_CORRUPT) { goto relocate; @@ -1633,35 +1634,17 @@ static int lfs_dir_compact(lfs_t *lfs, return err; } - // TODO huh? + // successful compaction, swap dir pair to indicate most recent LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0); + lfs_pair_swap(dir->pair); + dir->count = end - begin; + dir->off = commit.off; + dir->etag = commit.ptag; // update gstate lfs->gdelta = (lfs_gstate_t){0}; if (!relocated) { lfs->gdisk = lfs->gstate; } - - // TODO here?? - if (relocated) { - // update references if we relocated - LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, - oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - int err = lfs_fs_relocate(lfs, oldpair, dir->pair); - if (err) { - // TODO make better - //dir->pair[1] = oldpair[1]; // - return err; - } -// LFS_DEBUG("Relocated %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, -// oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); - } - - // successful compaction, swap dir pair to indicate most recent - lfs_pair_swap(dir->pair); - dir->rev = nrev; - dir->count = end - begin; - dir->off = commit.off; - dir->etag = commit.ptag; } break; @@ -1689,6 +1672,15 @@ static int lfs_dir_compact(lfs_t *lfs, continue; } + if (relocated) { + // update references if we relocated + LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); + int err = lfs_fs_relocate(lfs, oldpair, dir->pair); + if (err) { + return err; + } + } return 0; } @@ -1714,8 +1706,8 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } // calculate changes to the directory + lfs_mdir_t olddir = *dir; bool hasdelete = false; - lfs_mdir_t olddir = *dir; // TODO is this a good idea? for (int i = 0; i < attrcount; i++) { if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE) { dir->count += 1; @@ -1741,7 +1733,11 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } if (err != LFS_ERR_NOENT && pdir.split) { - return lfs_dir_drop(lfs, &pdir, dir); + err = lfs_dir_drop(lfs, &pdir, dir); + if (err) { + *dir = olddir; + return err; + } } } @@ -1837,9 +1833,8 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, // we need to copy the pair so they don't get clobbered if we refetch // our mdir. for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) { - if (&d->m != dir && lfs_pair_cmp(d->m.pair, dir->pair) == 0) { + if (&d->m != dir && lfs_pair_cmp(d->m.pair, olddir.pair) == 0) { d->m = *dir; - for (int i = 0; i < attrcount; i++) { if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE && d->id == lfs_tag_id(attrs[i].tag)) { @@ -1862,9 +1857,8 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, } } - lfs_block_t pair[2] = {dir->pair[0], dir->pair[1]}; for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) { - if (lfs_pair_cmp(d->m.pair, pair) == 0) { + if (lfs_pair_cmp(d->m.pair, olddir.pair) == 0) { while (d->id >= d->m.count && d->m.split) { // we split and id is on tail now d->id -= d->m.count; @@ -2714,66 +2708,52 @@ int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) { LFS_TRACE("lfs_file_sync(%p, %p)", (void*)lfs, (void*)file); LFS_ASSERT(file->flags & LFS_F_OPENED); - while (true) { - int err = lfs_file_flush(lfs, file); - if (err) { - file->flags |= LFS_F_ERRED; - LFS_TRACE("lfs_file_sync -> %d", err); - return err; - } - - if ((file->flags & LFS_F_DIRTY) && - !(file->flags & LFS_F_ERRED) && - !lfs_pair_isnull(file->m.pair)) { - // update dir entry - uint16_t type; - const void *buffer; - lfs_size_t size; - struct lfs_ctz ctz; - if (file->flags & LFS_F_INLINE) { - // inline the whole file - type = LFS_TYPE_INLINESTRUCT; - buffer = file->cache.buffer; - size = file->ctz.size; - } else { - // update the ctz reference - type = LFS_TYPE_CTZSTRUCT; - // copy ctz so alloc will work during a relocate - ctz = file->ctz; - lfs_ctz_tole32(&ctz); - buffer = &ctz; - size = sizeof(ctz); - } - - // commit file data and attributes - err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS( - {LFS_MKTAG(type, file->id, size), buffer}, - {LFS_MKTAG(LFS_FROM_USERATTRS, file->id, - file->cfg->attr_count), file->cfg->attrs})); - if (err) { -// if (err == LFS_ERR_NOSPC && (file->flags & LFS_F_INLINE)) { -// goto relocate; -// } - file->flags |= LFS_F_ERRED; - LFS_TRACE("lfs_file_sync -> %d", err); - return err; - } + int err = lfs_file_flush(lfs, file); + if (err) { + file->flags |= LFS_F_ERRED; + LFS_TRACE("lfs_file_sync -> %d", err); + return err; + } - file->flags &= ~LFS_F_DIRTY; + if ((file->flags & LFS_F_DIRTY) && + !(file->flags & LFS_F_ERRED) && + !lfs_pair_isnull(file->m.pair)) { + // update dir entry + uint16_t type; + const void *buffer; + lfs_size_t size; + struct lfs_ctz ctz; + if (file->flags & LFS_F_INLINE) { + // inline the whole file + type = LFS_TYPE_INLINESTRUCT; + buffer = file->cache.buffer; + size = file->ctz.size; + } else { + // update the ctz reference + type = LFS_TYPE_CTZSTRUCT; + // copy ctz so alloc will work during a relocate + ctz = file->ctz; + lfs_ctz_tole32(&ctz); + buffer = &ctz; + size = sizeof(ctz); } - LFS_TRACE("lfs_file_sync -> %d", 0); - return 0; - -relocate: - // inline file doesn't fit anymore - err = lfs_file_outline(lfs, file); + // commit file data and attributes + err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS( + {LFS_MKTAG(type, file->id, size), buffer}, + {LFS_MKTAG(LFS_FROM_USERATTRS, file->id, + file->cfg->attr_count), file->cfg->attrs})); if (err) { file->flags |= LFS_F_ERRED; LFS_TRACE("lfs_file_sync -> %d", err); return err; } + + file->flags &= ~LFS_F_DIRTY; } + + LFS_TRACE("lfs_file_sync -> %d", 0); + return 0; } lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file, @@ -3947,9 +3927,7 @@ static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2], // use fetchmatch with callback to find pairs parent->tail[0] = 0; parent->tail[1] = 1; - int i = 0; while (!lfs_pair_isnull(parent->tail)) { - i += 1; lfs_stag_t tag = lfs_dir_fetchmatch(lfs, parent, parent->tail, LFS_MKTAG(0x7ff, 0, 0x3ff), LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, 8), @@ -3957,7 +3935,6 @@ static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2], lfs_fs_parent_match, &(struct lfs_fs_parent_match){ lfs, {pair[0], pair[1]}}); if (tag && tag != LFS_ERR_NOENT) { - //printf("PARENT %d\n", i); return tag; } } @@ -4014,7 +3991,6 @@ static int lfs_fs_relocate(lfs_t *lfs, } } - //printf("parent %x %x\n", parent.pair[0], parent.pair[1]); lfs_pair_tole32(newpair); int err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS( {LFS_MKTAG_IF(moveid != 0x3ff, @@ -4049,7 +4025,6 @@ static int lfs_fs_relocate(lfs_t *lfs, } // replace bad pair, either we clean up desync, or no desync occured - //printf("pred %x %x\n", parent.pair[0], parent.pair[1]); lfs_pair_tole32(newpair); err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS( {LFS_MKTAG_IF(moveid != 0x3ff, diff --git a/tests/test_dirs.toml b/tests/test_dirs.toml index f3a819d1..270f4f8e 100644 --- a/tests/test_dirs.toml +++ b/tests/test_dirs.toml @@ -155,7 +155,7 @@ code = ''' ''' [[case]] # reentrant many directory creation/rename/removal -define.N = [5, 10] # TODO changed from 20, should we be able to do more? +define.N = [5, 11] reentrant = true code = ''' err = lfs_mount(&lfs, &cfg); diff --git a/tests/test_exhaustion.toml b/tests/test_exhaustion.toml index 0460071e..b89a69aa 100644 --- a/tests/test_exhaustion.toml +++ b/tests/test_exhaustion.toml @@ -338,7 +338,6 @@ exhausted: define.LFS_ERASE_CYCLES = 0xffffffff define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_BLOCK_CYCLES = [5, 4, 3, 2, 1] -#define.LFS_BLOCK_CYCLES = [4, 2] define.CYCLES = 100 define.FILES = 10 code = ''' From b69cf890e6889ed972ef6afa9dc4aaafb5060b73 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 9 Feb 2020 22:43:20 -0600 Subject: [PATCH 24/41] Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks. --- lfs.c | 55 +++++++++++++++++++++++++++---------------------------- 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/lfs.c b/lfs.c index 5ad81724..1cbcd33e 100644 --- a/lfs.c +++ b/lfs.c @@ -714,7 +714,7 @@ static int lfs_dir_traverse(lfs_t *lfs, uint16_t fromid = lfs_tag_size(tag); uint16_t toid = lfs_tag_id(tag); int err = lfs_dir_traverse(lfs, - buffer, 0, LFS_BLOCK_NULL, NULL, 0, + buffer, 0, 0xffffffff, NULL, 0, LFS_MKTAG(0x600, 0x3ff, 0), LFS_MKTAG(LFS_TYPE_STRUCT, 0, 0), fromid, fromid+1, toid-fromid+diff, @@ -774,7 +774,7 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, // now scan tags to fetch the actual dir and find possible match for (int i = 0; i < 2; i++) { lfs_off_t off = 0; - lfs_tag_t ptag = LFS_BLOCK_NULL; + lfs_tag_t ptag = 0xffffffff; uint16_t tempcount = 0; lfs_block_t temptail[2] = {LFS_BLOCK_NULL, LFS_BLOCK_NULL}; @@ -782,7 +782,7 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, lfs_stag_t tempbesttag = besttag; dir->rev = lfs_tole32(dir->rev); - uint32_t crc = lfs_crc(LFS_BLOCK_NULL, &dir->rev, sizeof(dir->rev)); + uint32_t crc = lfs_crc(0xffffffff, &dir->rev, sizeof(dir->rev)); dir->rev = lfs_fromle32(dir->rev); while (true) { @@ -853,7 +853,7 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, dir->split = tempsplit; // reset crc - crc = LFS_BLOCK_NULL; + crc = 0xffffffff; continue; } @@ -1231,14 +1231,14 @@ static int lfs_dir_commitattr(lfs_t *lfs, struct lfs_commit *commit, } static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { + const lfs_off_t off1 = commit->off; + const uint32_t crc1 = commit->crc; // align to program units - const lfs_off_t off1 = commit->off + sizeof(lfs_tag_t); - const lfs_off_t end = lfs_alignup(off1 + sizeof(uint32_t), + const lfs_off_t end = lfs_alignup(off1 + 2*sizeof(uint32_t), lfs->cfg->prog_size); - uint32_t ncrc = commit->crc; // create crc tags to fill up remainder of commit, note that - // padding is not crcd, which lets fetches skip padding but + // padding is not crced, which lets fetches skip padding but // makes committing a bit more complicated while (commit->off < end) { lfs_off_t off = commit->off + sizeof(lfs_tag_t); @@ -1248,7 +1248,7 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { } // read erased state from next program unit - lfs_tag_t tag = LFS_BLOCK_NULL; + lfs_tag_t tag = 0xffffffff; int err = lfs_bd_read(lfs, NULL, &lfs->rcache, sizeof(tag), commit->block, noff, &tag, sizeof(tag)); @@ -1272,10 +1272,9 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { return err; } - ncrc = commit->crc; commit->off += sizeof(tag)+lfs_tag_size(tag); commit->ptag = tag ^ ((lfs_tag_t)reset << 31); - commit->crc = LFS_BLOCK_NULL; // reset crc for next "commit" + commit->crc = 0xffffffff; // reset crc for next "commit" } // flush buffers @@ -1286,10 +1285,16 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { // successful commit, check checksums to make sure lfs_off_t off = commit->begin; - lfs_off_t noff = off1; + lfs_off_t noff = off1 + sizeof(uint32_t); while (off < end) { - uint32_t crc = LFS_BLOCK_NULL; + uint32_t crc = 0xffffffff; for (lfs_off_t i = off; i < noff+sizeof(uint32_t); i++) { + // check against written crc, may catch blocks that + // become readonly and match our commit size exactly + if (i == off1 && crc != crc1) { + return LFS_ERR_CORRUPT; + } + // leave it up to caching to make this efficient uint8_t dat; err = lfs_bd_read(lfs, @@ -1299,12 +1304,6 @@ static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) { return err; } - // check against written crc to detect if block is readonly - // (we may pick up old commits) - if (i == noff && crc != ncrc) { - return LFS_ERR_CORRUPT; - } - crc = lfs_crc(crc, &dat, 1); } @@ -1351,7 +1350,7 @@ static int lfs_dir_alloc(lfs_t *lfs, lfs_mdir_t *dir) { // set defaults dir->off = sizeof(dir->rev); - dir->etag = LFS_BLOCK_NULL; + dir->etag = 0xffffffff; dir->count = 0; dir->tail[0] = LFS_BLOCK_NULL; dir->tail[1] = LFS_BLOCK_NULL; @@ -1445,7 +1444,7 @@ static int lfs_dir_compact(lfs_t *lfs, // find size lfs_size_t size = 0; int err = lfs_dir_traverse(lfs, - source, 0, LFS_BLOCK_NULL, attrs, attrcount, + source, 0, 0xffffffff, attrs, attrcount, LFS_MKTAG(0x400, 0x3ff, 0), LFS_MKTAG(LFS_TYPE_NAME, 0, 0), begin, end, -begin, @@ -1540,8 +1539,8 @@ static int lfs_dir_compact(lfs_t *lfs, struct lfs_commit commit = { .block = dir->pair[1], .off = 0, - .ptag = LFS_BLOCK_NULL, - .crc = LFS_BLOCK_NULL, + .ptag = 0xffffffff, + .crc = 0xffffffff, .begin = 0, .end = lfs->cfg->block_size - 8, @@ -1570,7 +1569,7 @@ static int lfs_dir_compact(lfs_t *lfs, // traverse the directory, this time writing out all unique tags err = lfs_dir_traverse(lfs, - source, 0, LFS_BLOCK_NULL, attrs, attrcount, + source, 0, 0xffffffff, attrs, attrcount, LFS_MKTAG(0x400, 0x3ff, 0), LFS_MKTAG(LFS_TYPE_NAME, 0, 0), begin, end, -begin, @@ -1747,7 +1746,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, .block = dir->pair[0], .off = dir->off, .ptag = dir->etag, - .crc = LFS_BLOCK_NULL, + .crc = 0xffffffff, .begin = dir->off, .end = lfs->cfg->block_size - 8, @@ -3445,7 +3444,7 @@ static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) { LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->cache_size == 0); // check that the block size is large enough to fit ctz pointers - LFS_ASSERT(4*lfs_npw2(LFS_BLOCK_NULL / (lfs->cfg->block_size-2*4)) + LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4)) <= lfs->cfg->block_size); // block_cycles = 0 is no longer supported. @@ -4370,7 +4369,7 @@ static int lfs1_dir_fetch(lfs_t *lfs, continue; } - uint32_t crc = LFS_BLOCK_NULL; + uint32_t crc = 0xffffffff; lfs1_dir_tole32(&test); lfs1_crc(&crc, &test, sizeof(test)); lfs1_dir_fromle32(&test); @@ -4806,7 +4805,7 @@ int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) { dir2.pair[1] = dir1.pair[1]; dir2.rev = dir1.d.rev; dir2.off = sizeof(dir2.rev); - dir2.etag = LFS_BLOCK_NULL; + dir2.etag = 0xffffffff; dir2.count = 0; dir2.tail[0] = lfs->lfs1->root[0]; dir2.tail[1] = lfs->lfs1->root[1]; From 9f546f154fe8b7177b253db94a763dadaf4459df Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 9 Feb 2020 23:09:46 -0600 Subject: [PATCH 25/41] Updated .travis.yml and added additional geometry constraints Moved .travis.yml over to use the new test framework. A part of this involved testing all of the configurations ran on the old framework and deciding which to carry over. The new framework duplicates some of the cases tested by the configurations so some configurations could be dropped. The .travis.yml includes some extreme ones, such as no inline files, relocations every cycle, no intrinsics, power-loss every byte, unaligned block_count and lookahead, and odd read_sizes. There were several configurations were some tests failed because of limitations in the tests themselves, so many conditions were added to make sure the configurations can run on as many tests as possible. --- .travis.yml | 57 ++++++++++++++++++++++---------------- scripts/test.py | 18 ++++++------ tests/test_alloc.toml | 11 ++++---- tests/test_badblocks.toml | 3 ++ tests/test_entries.toml | 2 +- tests/test_exhaustion.toml | 1 + tests/test_move.toml | 4 +++ tests/test_orphans.toml | 1 + tests/test_truncate.toml | 2 +- 9 files changed, 60 insertions(+), 39 deletions(-) diff --git a/.travis.yml b/.travis.yml index 42e59dc5..a4697b1f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,6 +2,7 @@ env: global: - CFLAGS=-Werror + - MAKEFLAGS=-j # Common test script script: @@ -15,32 +16,27 @@ script: -include stdio.h" # run tests - - make test QUIET=1 + - make clean test TFLAGS+="-nrk" - # run tests with a few different configurations - - make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=1 -DLFS_CACHE_SIZE=4" - - make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=512 -DLFS_CACHE_SIZE=512 -DLFS_BLOCK_CYCLES=16" - - make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=8 -DLFS_CACHE_SIZE=16 -DLFS_BLOCK_CYCLES=2" - - make test QUIET=1 CFLAGS+="-DLFS_BLOCK_COUNT=1023 -DLFS_LOOKAHEAD_SIZE=256" + # common real-life geometries + # NOR flash: read/prog = 1 block = 4KiB + - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=1 -DLFS_BLOCK_SIZE=4096" + # SD card: read/prog = 512 block = 512 + - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=512 -DLFS_BLOCK_SIZE=512" + # NAND flash: read/prog = 4KiB block = 32KiB + - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=4096 -DLFS_BLOCK_SIZE=\(32*1024\)" - - make clean test QUIET=1 CFLAGS+="-DLFS_INLINE_MAX=0" - - make clean test QUIET=1 CFLAGS+="-DLFS_EMUBD_ERASE_VALUE=0xff" - - make clean test QUIET=1 CFLAGS+="-DLFS_NO_INTRINSICS" - - # additional configurations that don't support all tests (this should be - # fixed but at the moment it is what it is) - - make test_files QUIET=1 - CFLAGS+="-DLFS_READ_SIZE=1 -DLFS_BLOCK_SIZE=4096" - - make test_files QUIET=1 - CFLAGS+="-DLFS_READ_SIZE=\(2*1024\) -DLFS_BLOCK_SIZE=\(64*1024\)" - - make test_files QUIET=1 - CFLAGS+="-DLFS_READ_SIZE=\(8*1024\) -DLFS_BLOCK_SIZE=\(64*1024\)" - - make test_files QUIET=1 - CFLAGS+="-DLFS_READ_SIZE=11 -DLFS_BLOCK_SIZE=704" + # other extreme geometries for corner cases + - make clean test TFLAGS+="-nrk -DLFS_NO_INTRINSICS" + - make clean test TFLAGS+="-nrk -DLFS_INLINE_MAX=0" + - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=1 -DLFS_CACHE_SIZE=1" + - make clean test TFLAGS+="-nrk -DLFS_BLOCK_CYCLES=1" + - make clean test TFLAGS+="-nrk -DLFS_BLOCK_COUNT=1023 -DLFS_LOOKAHEAD_SIZE=256" + - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=11 -DLFS_BLOCK_SIZE=704" # compile and find the code size with the smallest configuration - make clean size - OBJ="$(ls lfs*.o | tr '\n' ' ')" + OBJ="$(ls lfs*.c | sed 's/\.c/\.o/' | tr '\n' ' ')" CFLAGS+="-DLFS_NO_ASSERT -DLFS_NO_DEBUG -DLFS_NO_WARN -DLFS_NO_ERROR" | tee sizes @@ -77,7 +73,7 @@ jobs: - STAGE=test - NAME=littlefs-arm - CC="arm-linux-gnueabi-gcc --static -mthumb" - - EXEC="qemu-arm" + - TFLAGS="-e qemu-arm" install: - sudo apt-get install gcc-arm-linux-gnueabi @@ -92,7 +88,7 @@ jobs: - STAGE=test - NAME=littlefs-powerpc - CC="powerpc-linux-gnu-gcc --static" - - EXEC="qemu-ppc" + - TFLAGS="-e qemu-ppc" install: - sudo apt-get install gcc-powerpc-linux-gnu @@ -107,7 +103,7 @@ jobs: - STAGE=test - NAME=littlefs-mips - CC="mips-linux-gnu-gcc --static" - - EXEC="qemu-mips" + - TFLAGS="-e qemu-mips" install: - sudo apt-get install gcc-mips-linux-gnu @@ -116,6 +112,16 @@ jobs: - mips-linux-gnu-gcc --version - qemu-mips -version + # test under valgrind, checking for memory errors + - stage: test + env: + - STAGE=test + - NAME=littlefs-valgrind + - TFLAGS="--valgrind" + install: + - sudo apt-get install valgrind + - valgrind --version + # self-host with littlefs-fuse for fuzz test - stage: test env: @@ -288,6 +294,9 @@ before_install: \"description\": \"${STATUS:-In progress}\", \"target_url\": \"https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$TRAVIS_JOB_ID\" }" + - sudo apt-get install python3-pip + - sudo pip3 install toml + after_failure: - | diff --git a/scripts/test.py b/scripts/test.py index cbc8838f..c662a76b 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -184,7 +184,8 @@ def shouldtest(self, **args): elif self.if_ is not None: if_ = self.if_ while True: - for k, v in self.defines.items(): + for k, v in sorted(self.defines.items(), + key=lambda x: len(x[0]), reverse=True): if k in if_: if_ = if_.replace(k, '(%s)' % v) break @@ -295,11 +296,11 @@ def shouldtest(self, **args): return not self.leaky and super().shouldtest(**args) def test(self, exec=[], **args): - exec = exec + [ + exec = [ 'valgrind', '--leak-check=full', '--error-exitcode=4', - '-q'] + '-q'] + exec return super().test(exec=exec, **args) class ReentrantTestCase(TestCase): @@ -310,7 +311,7 @@ def __init__(self, config, **args): def shouldtest(self, **args): return self.reentrant and super().shouldtest(**args) - def test(self, exec=[], persist=False, gdb=False, failure=None, **args): + def test(self, persist=False, gdb=False, failure=None, **args): for cycles in it.count(1): # clear disk first? if cycles == 1 and persist != 'noerase': @@ -376,10 +377,11 @@ def __init__(self, path, classes=[TestCase], defines={}, # code lineno? if 'code' in case: case['code_lineno'] = code_linenos.pop() - # give our case's config a copy of our "global" config - for k, v in config.items(): - if k not in case: - case[k] = v + # merge conditions if necessary + if 'if' in config and 'if' in case: + case['if'] = '(%s) && (%s)' % (config['if'], case['if']) + elif 'if' in config: + case['if'] = config['if'] # initialize test case self.cases.append(TestCase(case, filter=filter, suite=self, caseno=i+1, lineno=lineno, **args)) diff --git a/tests/test_alloc.toml b/tests/test_alloc.toml index f15a7461..c05f001c 100644 --- a/tests/test_alloc.toml +++ b/tests/test_alloc.toml @@ -1,9 +1,10 @@ # allocator tests -# note for these to work there are many constraints on the device geometry +# note for these to work there are a number constraints on the device geometry +if = 'LFS_BLOCK_CYCLES == -1' [[case]] # parallel allocation test define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-6)) / FILES)' code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; lfs_file_t files[FILES]; @@ -46,7 +47,7 @@ code = ''' [[case]] # serial allocation test define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-6)) / FILES)' code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; @@ -85,7 +86,7 @@ code = ''' [[case]] # parallel allocation reuse test define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-6)) / FILES)' define.CYCLES = [1, 10] code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; @@ -140,7 +141,7 @@ code = ''' [[case]] # serial allocation reuse test define.FILES = 3 -define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-4)) / FILES)' +define.SIZE = '(((LFS_BLOCK_SIZE-8)*(LFS_BLOCK_COUNT-6)) / FILES)' define.CYCLES = [1, 10] code = ''' const char *names[FILES] = {"bacon", "eggs", "pancakes"}; diff --git a/tests/test_badblocks.toml b/tests/test_badblocks.toml index 44c3e097..06967a67 100644 --- a/tests/test_badblocks.toml +++ b/tests/test_badblocks.toml @@ -1,3 +1,6 @@ +# bad blocks with block cycles should be tested in test_relocations +if = 'LFS_BLOCK_CYCLES == -1' + [[case]] # single bad blocks define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_ERASE_CYCLES = 0xffffffff diff --git a/tests/test_entries.toml b/tests/test_entries.toml index cbbf6a1d..81e175f5 100644 --- a/tests/test_entries.toml +++ b/tests/test_entries.toml @@ -3,7 +3,7 @@ # still pass with other inline sizes but wouldn't be testing anything. define.LFS_CACHE_SIZE = 512 -if = 'LFS_CACHE_SIZE == 512' +if = 'LFS_CACHE_SIZE % LFS_PROG_SIZE == 0 && LFS_CACHE_SIZE == 512' [[case]] # entry grow test code = ''' diff --git a/tests/test_exhaustion.toml b/tests/test_exhaustion.toml index b89a69aa..26eeeba2 100644 --- a/tests/test_exhaustion.toml +++ b/tests/test_exhaustion.toml @@ -340,6 +340,7 @@ define.LFS_BLOCK_COUNT = 256 # small bd so test runs faster define.LFS_BLOCK_CYCLES = [5, 4, 3, 2, 1] define.CYCLES = 100 define.FILES = 10 +if = 'LFS_BLOCK_CYCLES < CYCLES/10' code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; diff --git a/tests/test_move.toml b/tests/test_move.toml index f290ed4a..bb3b713f 100644 --- a/tests/test_move.toml +++ b/tests/test_move.toml @@ -148,6 +148,7 @@ code = ''' [[case]] # move file corrupt source and dest in = "lfs.c" +if = 'LFS_PROG_SIZE <= 0x3fe' # only works with one crc per commit code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -239,6 +240,7 @@ code = ''' [[case]] # move file after corrupt in = "lfs.c" +if = 'LFS_PROG_SIZE <= 0x3fe' # only works with one crc per commit code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -593,6 +595,7 @@ code = ''' [[case]] # move dir corrupt source and dest in = "lfs.c" +if = 'LFS_PROG_SIZE <= 0x3fe' # only works with one crc per commit code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; @@ -692,6 +695,7 @@ code = ''' [[case]] # move dir after corrupt in = "lfs.c" +if = 'LFS_PROG_SIZE <= 0x3fe' # only works with one crc per commit code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; diff --git a/tests/test_orphans.toml b/tests/test_orphans.toml index 8d0f3deb..241e273e 100644 --- a/tests/test_orphans.toml +++ b/tests/test_orphans.toml @@ -1,5 +1,6 @@ [[case]] # orphan test in = "lfs.c" +if = 'LFS_PROG_SIZE <= 0x3fe' # only works with one crc per commit code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; diff --git a/tests/test_truncate.toml b/tests/test_truncate.toml index 44169862..c11285b7 100644 --- a/tests/test_truncate.toml +++ b/tests/test_truncate.toml @@ -100,7 +100,7 @@ code = ''' lfs_file_open(&lfs, &file, "sequence", LFS_O_RDWR | LFS_O_CREAT | LFS_O_TRUNC) => 0; - size = lfs.cfg->cache_size; + size = lfs_min(lfs.cfg->cache_size, sizeof(buffer)/2); lfs_size_t qsize = size / 4; uint8_t *wb = buffer; uint8_t *rb = buffer + size; From f4b17b379c2d7ce242928d626d942a941703113a Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Wed, 12 Feb 2020 10:48:54 -0600 Subject: [PATCH 26/41] Added test.py support for tmpfs-backed disks RAM-backed testing is faster than file-backed testing. This is why test.py uses rambd by default. So why add support for tmpfs-backed disks if we can already run tests in RAM? For reentrant testing. Under reentrant testing we simulate power-loss by forcefully exiting the test program at specific times. To make this power-loss meaningful, we need to persist the disk across these power-losses. However, it's interesting to note this persistence doesn't need to be actually backed by the filesystem. It may be possible to rearchitecture the tests to simulate power-loss a different way, by say, using coroutines or setjmp/longjmp to leave behind ongoing filesystem operations without terminating the program completely. But at this point, I think it's best to work with what we have. And simply putting the test disks into a tmpfs mount-point seems to work just fine. Note this does force serialization of the tests, which isn't required otherwise. Currently they are only serialized due to limitations in test.py. If a future change wants to perallelize the tests, it may need to rework RAM-backed reentrant tests. --- .travis.yml | 6 ++++++ scripts/test.py | 15 +++++++++------ 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index a4697b1f..16d3f10b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -15,6 +15,12 @@ script: -Duser_provided_block_device_sync=NULL -include stdio.h" + # setup a ram-backed disk to speed up reentrant tests + - | + mkdir disks + sudo mount -t tmpfs -o size=100m tmpfs disks + export TFLAGS="$TFLAGS --disk=disks/disk" + # run tests - make clean test TFLAGS+="-nrk" diff --git a/scripts/test.py b/scripts/test.py index c662a76b..018d0b23 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -200,22 +200,25 @@ def shouldtest(self, **args): return True def test(self, exec=[], persist=False, cycles=None, - gdb=False, failure=None, **args): + gdb=False, failure=None, disk=None, **args): # build command cmd = exec + ['./%s.test' % self.suite.path, repr(self.caseno), repr(self.permno)] # persist disk or keep in RAM for speed? if persist: + if not disk: + disk = self.suite.path + '.disk' if persist != 'noerase': try: - os.remove(self.suite.path + '.disk') + with open(disk, 'w') as f: + f.truncate(0) if args.get('verbose', False): - print('rm', self.suite.path + '.disk') + print('truncate --size=0', disk) except FileNotFoundError: pass - cmd.append(self.suite.path + '.disk') + cmd.append(disk) # simulate power-loss after n cycles? if cycles: @@ -704,8 +707,6 @@ def main(**args): stdout = perm.result.stdout[:-1] else: stdout = perm.result.stdout - if (not args.get('verbose', False) and len(stdout) > 5): - sys.stdout.write('...\n') for line in stdout[-5:]: sys.stdout.write(line) if perm.result.assert_: @@ -766,4 +767,6 @@ def main(**args): help="Run non-leaky tests under valgrind to check for memory leaks.") parser.add_argument('-e', '--exec', default=[], type=lambda e: e.split(' '), help="Run tests with another executable prefixed on the command line.") + parser.add_argument('-d', '--disk', + help="Specify a file to use for persistent/reentrant tests.") sys.exit(main(**vars(parser.parse_args()))) From dcae185a000576fc91c12287149e65b7588e925e Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Wed, 12 Feb 2020 11:31:34 -0600 Subject: [PATCH 27/41] Fixed typo in LFS_MKTAG_IF_ELSE --- lfs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lfs.c b/lfs.c index 1cbcd33e..354cebc7 100644 --- a/lfs.c +++ b/lfs.c @@ -269,7 +269,7 @@ typedef int32_t lfs_stag_t; ((cond) ? LFS_MKTAG(type, id, size) : LFS_MKTAG(LFS_FROM_NOOP, 0, 0)) #define LFS_MKTAG_IF_ELSE(cond, type1, id1, size1, type2, id2, size2) \ - ((cond) ? LFS_MKTAG(type, id, size) : LFS_MKTAG(type2, id2, size2)) + ((cond) ? LFS_MKTAG(type1, id1, size1) : LFS_MKTAG(type2, id2, size2)) static inline bool lfs_tag_isvalid(lfs_tag_t tag) { return !(tag & 0x80000000); From c7987a31626a4c6694bdf3492e0ade7f9884677c Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 16 Feb 2020 15:11:20 -0600 Subject: [PATCH 28/41] Restructured .travis.yml to span more jobs The core of littlefs's CI testing is the full test suite, `make test`, run under a number of configurations: - Processor architecture: - x86 (native) - Arm Thumb - MIPS - PowerPC - Storage geometry: - rs=16 ps=16 cs=64 bs=512 (default) - rs=1 ps=1 cs=64 bs=4KiB (NOR flash) - rs=512 ps=512 cs=512 bs=512 (eMMC) - rs=4KiB ps=4KiB cs=4KiB bs=32KiB (NAND flash) - Other corner cases: - no intrinsics - no inline - byte-level read/writes - single block-cycles - odd block counts - odd block sizes The number of different configurations we need to test quickly exceeds the 50 minute time limit Travis has on jobs. Fortunately, we can split these tests out into multiple jobs. This seems to be the intended course of action for large CI "builds" in Travis, as this gives Travis a finer grain of control over limiting builds. Unfortunately, this created a couple issues: 1. The Travis configuration isn't actually that flexible. It allows a single "matrix expansion" which can be generated from top-level lists of different configurations. But it doesn't let you generate a matrix from two seperate environment variable lists (for arch + geometry). Without multiple matrix expansions, we're stuck writing out each test permutation by hand. On the bright-side, this was a good chance to really learn how YAML anchors work. I'm torn because on one hand anchors add what feels like unnecessary complexity to a config language, on the other hand, they did help quite a bit in working around Travis's limitations. 2. Now that we have 47 jobs instead of 7, reporting a separate status for each job stops making sense. What I've opted for here is to use a special NAME variable to deduplicate jobs, and used a few state-less rules to hopefully have the reported status make sense most of the time. - Overwrite "pending" statuses so that the last job to start owns the most recent "pending" status - Don't overwrite "failure" statuses unless the job number matches our own (in the case of CI restarts) - Don't write "success" statuses unless the job number matches our own, this should delay a green check-mark until the last-to-start job finishes - Always overwrite non-failures with "failure" statuses This does mean a temporary "success" may appear if the last job terminates before earlier jobs. But this is the simpliest solution I can think of without storing some complex state somewhere. Note we can only report the size this way because it's cheap to calculate in every job. --- .travis.yml | 638 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 366 insertions(+), 272 deletions(-) diff --git a/.travis.yml b/.travis.yml index 16d3f10b..5d10d4ee 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,51 +1,64 @@ -# Environment variables +# environment variables env: global: - CFLAGS=-Werror - MAKEFLAGS=-j -# Common test script -script: +# common installation +_: &install-common + # need toml, also pip3 isn't installed by default? + - sudo apt-get install python3-pip + - sudo pip3 install toml + # setup a ram-backed disk to speed up reentrant tests + - mkdir disks + - sudo mount -t tmpfs -o size=100m tmpfs disks + - export TFLAGS="$TFLAGS --disk=disks/disk" + +# test cases +_: &test-example # make sure example can at least compile - - sed -n '/``` c/,/```/{/```/d; p;}' README.md > test.c && + - sed -n '/``` c/,/```/{/```/d; p}' README.md > test.c && make all CFLAGS+=" -Duser_provided_block_device_read=NULL -Duser_provided_block_device_prog=NULL -Duser_provided_block_device_erase=NULL -Duser_provided_block_device_sync=NULL -include stdio.h" - - # setup a ram-backed disk to speed up reentrant tests - - | - mkdir disks - sudo mount -t tmpfs -o size=100m tmpfs disks - export TFLAGS="$TFLAGS --disk=disks/disk" - - # run tests - - make clean test TFLAGS+="-nrk" - - # common real-life geometries +# default tests +_: &test-default + # normal+reentrant tests + - make test TFLAGS+="-nrk" +# common real-life geometries +_: &test-nor # NOR flash: read/prog = 1 block = 4KiB - - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=1 -DLFS_BLOCK_SIZE=4096" - # SD card: read/prog = 512 block = 512 - - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=512 -DLFS_BLOCK_SIZE=512" + - make test TFLAGS+="-nrk -DLFS_READ_SIZE=1 -DLFS_BLOCK_SIZE=4096" +_: &test-emmc + # eMMC: read/prog = 512 block = 512 + - make test TFLAGS+="-nrk -DLFS_READ_SIZE=512 -DLFS_BLOCK_SIZE=512" +_: &test-nand # NAND flash: read/prog = 4KiB block = 32KiB - - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=4096 -DLFS_BLOCK_SIZE=\(32*1024\)" - - # other extreme geometries for corner cases - - make clean test TFLAGS+="-nrk -DLFS_NO_INTRINSICS" - - make clean test TFLAGS+="-nrk -DLFS_INLINE_MAX=0" - - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=1 -DLFS_CACHE_SIZE=1" - - make clean test TFLAGS+="-nrk -DLFS_BLOCK_CYCLES=1" - - make clean test TFLAGS+="-nrk -DLFS_BLOCK_COUNT=1023 -DLFS_LOOKAHEAD_SIZE=256" - - make clean test TFLAGS+="-nrk -DLFS_READ_SIZE=11 -DLFS_BLOCK_SIZE=704" + - make test TFLAGS+="-nrk -DLFS_READ_SIZE=4096 -DLFS_BLOCK_SIZE=\(32*1024\)" +# other extreme geometries that are useful for testing various corner cases +_: &test-no-intrinsics + - make test TFLAGS+="-nrk -DLFS_NO_INTRINSICS" +_: &test-no-inline + - make test TFLAGS+="-nrk -DLFS_INLINE_MAX=0" +_: &test-byte-writes + - make test TFLAGS+="-nrk -DLFS_READ_SIZE=1 -DLFS_CACHE_SIZE=1" +_: &test-block-cycles + - make test TFLAGS+="-nrk -DLFS_BLOCK_CYCLES=1" +_: &test-odd-block-count + - make test TFLAGS+="-nrk -DLFS_BLOCK_COUNT=1023 -DLFS_LOOKAHEAD_SIZE=256" +_: &test-odd-block-size + - make test TFLAGS+="-nrk -DLFS_READ_SIZE=11 -DLFS_BLOCK_SIZE=704" +# report size +_: &report-size # compile and find the code size with the smallest configuration - - make clean size + - make -j1 clean size OBJ="$(ls lfs*.c | sed 's/\.c/\.o/' | tr '\n' ' ')" CFLAGS+="-DLFS_NO_ASSERT -DLFS_NO_DEBUG -DLFS_NO_WARN -DLFS_NO_ERROR" | tee sizes - # update status if we succeeded, compare with master if possible - | if [ "$TRAVIS_TEST_RESULT" -eq 0 ] @@ -53,10 +66,10 @@ script: CURR=$(tail -n1 sizes | awk '{print $1}') PREV=$(curl -u "$GEKY_BOT_STATUSES" https://api.github.com/repos/$TRAVIS_REPO_SLUG/status/master \ | jq -re "select(.sha != \"$TRAVIS_COMMIT\") - | .statuses[] | select(.context == \"$STAGE/$NAME\").description + | .statuses[] | select(.context == \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\").description | capture(\"code size is (?[0-9]+)\").size" \ || echo 0) - + STATUS="Passed, code size is ${CURR}B" if [ "$PREV" -ne 0 ] then @@ -64,270 +77,351 @@ script: fi fi -# CI matrix +# stage control +stages: + - name: test + - name: deploy + if: branch = master AND type = push + +# job control jobs: - include: - # native testing - - stage: test - env: - - STAGE=test - - NAME=littlefs-x86 + # native testing + - &x86 + stage: test + env: + - NAME=littlefs-x86 + install: *install-common + script: [*test-example, *report-size] + - {<<: *x86, script: [*test-default, *report-size]} + - {<<: *x86, script: [*test-nor, *report-size]} + - {<<: *x86, script: [*test-emmc, *report-size]} + - {<<: *x86, script: [*test-nand, *report-size]} + - {<<: *x86, script: [*test-no-intrinsics, *report-size]} + - {<<: *x86, script: [*test-no-inline, *report-size]} + - {<<: *x86, script: [*test-byte-writes, *report-size]} + - {<<: *x86, script: [*test-block-cycles, *report-size]} + - {<<: *x86, script: [*test-odd-block-count, *report-size]} + - {<<: *x86, script: [*test-odd-block-size, *report-size]} + + # cross-compile with ARM (thumb mode) + - &arm + stage: test + env: + - NAME=littlefs-arm + - CC="arm-linux-gnueabi-gcc --static -mthumb" + - TFLAGS="$TFLAGS -e qemu-arm" + install: + - *install-common + - sudo apt-get install + gcc-arm-linux-gnueabi + libc6-dev-armel-cross + qemu-user + - arm-linux-gnueabi-gcc --version + - qemu-arm -version + script: [*test-example, *report-size] + - {<<: *arm, script: [*test-default, *report-size]} + - {<<: *arm, script: [*test-nor, *report-size]} + - {<<: *arm, script: [*test-emmc, *report-size]} + - {<<: *arm, script: [*test-nand, *report-size]} + - {<<: *arm, script: [*test-no-intrinsics, *report-size]} + - {<<: *arm, script: [*test-no-inline, *report-size]} + - {<<: *arm, script: [*test-byte-writes, *report-size]} + - {<<: *arm, script: [*test-block-cycles, *report-size]} + - {<<: *arm, script: [*test-odd-block-count, *report-size]} + - {<<: *arm, script: [*test-odd-block-size, *report-size]} - # cross-compile with ARM (thumb mode) - - stage: test - env: - - STAGE=test - - NAME=littlefs-arm - - CC="arm-linux-gnueabi-gcc --static -mthumb" - - TFLAGS="-e qemu-arm" - install: - - sudo apt-get install - gcc-arm-linux-gnueabi - libc6-dev-armel-cross - qemu-user - - arm-linux-gnueabi-gcc --version - - qemu-arm -version + # cross-compile with MIPS + - &mips + stage: test + env: + - NAME=littlefs-mips + - CC="mips-linux-gnu-gcc --static" + - TFLAGS="$TFLAGS -e qemu-mips" + install: + - *install-common + - sudo apt-get install + gcc-mips-linux-gnu + libc6-dev-mips-cross + qemu-user + - mips-linux-gnu-gcc --version + - qemu-mips -version + script: [*test-example, *report-size] + - {<<: *mips, script: [*test-default, *report-size]} + - {<<: *mips, script: [*test-nor, *report-size]} + - {<<: *mips, script: [*test-emmc, *report-size]} + - {<<: *mips, script: [*test-nand, *report-size]} + - {<<: *mips, script: [*test-no-intrinsics, *report-size]} + - {<<: *mips, script: [*test-no-inline, *report-size]} + - {<<: *mips, script: [*test-byte-writes, *report-size]} + - {<<: *mips, script: [*test-block-cycles, *report-size]} + - {<<: *mips, script: [*test-odd-block-count, *report-size]} + - {<<: *mips, script: [*test-odd-block-size, *report-size]} - # cross-compile with PowerPC - - stage: test - env: - - STAGE=test - - NAME=littlefs-powerpc - - CC="powerpc-linux-gnu-gcc --static" - - TFLAGS="-e qemu-ppc" - install: - - sudo apt-get install - gcc-powerpc-linux-gnu - libc6-dev-powerpc-cross - qemu-user - - powerpc-linux-gnu-gcc --version - - qemu-ppc -version + # cross-compile with PowerPC + - &powerpc + stage: test + env: + - NAME=littlefs-powerpc + - CC="powerpc-linux-gnu-gcc --static" + - TFLAGS="$TFLAGS -e qemu-ppc" + install: + - *install-common + - sudo apt-get install + gcc-powerpc-linux-gnu + libc6-dev-powerpc-cross + qemu-user + - powerpc-linux-gnu-gcc --version + - qemu-ppc -version + script: [*test-example, *report-size] + - {<<: *powerpc, script: [*test-default, *report-size]} + - {<<: *powerpc, script: [*test-nor, *report-size]} + - {<<: *powerpc, script: [*test-emmc, *report-size]} + - {<<: *powerpc, script: [*test-nand, *report-size]} + - {<<: *powerpc, script: [*test-no-intrinsics, *report-size]} + - {<<: *powerpc, script: [*test-no-inline, *report-size]} + - {<<: *powerpc, script: [*test-byte-writes, *report-size]} + - {<<: *powerpc, script: [*test-block-cycles, *report-size]} + - {<<: *powerpc, script: [*test-odd-block-count, *report-size]} + - {<<: *powerpc, script: [*test-odd-block-size, *report-size]} - # cross-compile with MIPS - - stage: test - env: - - STAGE=test - - NAME=littlefs-mips - - CC="mips-linux-gnu-gcc --static" - - TFLAGS="-e qemu-mips" - install: - - sudo apt-get install - gcc-mips-linux-gnu - libc6-dev-mips-cross - qemu-user - - mips-linux-gnu-gcc --version - - qemu-mips -version + # test under valgrind, checking for memory errors + - &valgrind + stage: test + env: + - NAME=littlefs-valgrind + - TFLAGS="$TFLAGS --valgrind" + install: + - *install-common + - sudo apt-get install valgrind + - valgrind --version + script: *test-example + - {<<: *valgrind, script: *test-default} + - {<<: *valgrind, script: *test-nor} + - {<<: *valgrind, script: *test-emmc} + - {<<: *valgrind, script: *test-nand} + - {<<: *valgrind, script: *test-no-intrinsics} + - {<<: *valgrind, script: *test-no-inline} + - {<<: *valgrind, script: *test-byte-writes} + - {<<: *valgrind, script: *test-block-cycles} + - {<<: *valgrind, script: *test-odd-block-count} + - {<<: *valgrind, script: *test-odd-block-size} - # test under valgrind, checking for memory errors - - stage: test - env: - - STAGE=test - - NAME=littlefs-valgrind - - TFLAGS="--valgrind" - install: - - sudo apt-get install valgrind - - valgrind --version + # self-host with littlefs-fuse for fuzz test + - stage: test + env: + - NAME=littlefs-fuse + if: branch !~ -prefix$ + install: + - *install-common + - sudo apt-get install libfuse-dev + - git clone --depth 1 https://github.com/geky/littlefs-fuse -b v2 + - fusermount -V + - gcc --version - # self-host with littlefs-fuse for fuzz test - - stage: test - env: - - STAGE=test - - NAME=littlefs-fuse - if: branch !~ -prefix$ - install: - - sudo apt-get install libfuse-dev - - git clone --depth 1 https://github.com/geky/littlefs-fuse -b v2 - - fusermount -V - - gcc --version - before_script: - # setup disk for littlefs-fuse - - rm -rf littlefs-fuse/littlefs/* - - cp -r $(git ls-tree --name-only HEAD) littlefs-fuse/littlefs + # setup disk for littlefs-fuse + - rm -rf littlefs-fuse/littlefs/* + - cp -r $(git ls-tree --name-only HEAD) littlefs-fuse/littlefs - - mkdir mount - - sudo chmod a+rw /dev/loop0 - - dd if=/dev/zero bs=512 count=4096 of=disk - - losetup /dev/loop0 disk - script: - # self-host test - - make -C littlefs-fuse + - mkdir mount + - sudo chmod a+rw /dev/loop0 + - dd if=/dev/zero bs=512 count=4096 of=disk + - losetup /dev/loop0 disk + script: + # self-host test + - make -C littlefs-fuse - - littlefs-fuse/lfs --format /dev/loop0 - - littlefs-fuse/lfs /dev/loop0 mount + - littlefs-fuse/lfs --format /dev/loop0 + - littlefs-fuse/lfs /dev/loop0 mount - - ls mount - - mkdir mount/littlefs - - cp -r $(git ls-tree --name-only HEAD) mount/littlefs - - cd mount/littlefs - - stat . - - ls -flh - - make -B test_dirs test_files QUIET=1 + - ls mount + - mkdir mount/littlefs + - cp -r $(git ls-tree --name-only HEAD) mount/littlefs + - cd mount/littlefs + - stat . + - ls -flh + - make -B test_dirs test_files QUIET=1 - # self-host with littlefs-fuse for fuzz test - - stage: test - env: - - STAGE=test - - NAME=littlefs-migration - if: branch !~ -prefix$ - install: - - sudo apt-get install libfuse-dev - - git clone --depth 1 https://github.com/geky/littlefs-fuse -b v2 v2 - - git clone --depth 1 https://github.com/geky/littlefs-fuse -b v1 v1 - - fusermount -V - - gcc --version - before_script: - # setup disk for littlefs-fuse - - rm -rf v2/littlefs/* - - cp -r $(git ls-tree --name-only HEAD) v2/littlefs + # test migration using littlefs-fuse + - stage: test + env: + - NAME=littlefs-migration + if: branch !~ -prefix$ + install: + - *install-common + - sudo apt-get install libfuse-dev + - git clone --depth 1 https://github.com/geky/littlefs-fuse -b v2 v2 + - git clone --depth 1 https://github.com/geky/littlefs-fuse -b v1 v1 + - fusermount -V + - gcc --version - - mkdir mount - - sudo chmod a+rw /dev/loop0 - - dd if=/dev/zero bs=512 count=4096 of=disk - - losetup /dev/loop0 disk - script: - # compile v1 and v2 - - make -C v1 - - make -C v2 + # setup disk for littlefs-fuse + - rm -rf v2/littlefs/* + - cp -r $(git ls-tree --name-only HEAD) v2/littlefs - # run self-host test with v1 - - v1/lfs --format /dev/loop0 - - v1/lfs /dev/loop0 mount + - mkdir mount + - sudo chmod a+rw /dev/loop0 + - dd if=/dev/zero bs=512 count=4096 of=disk + - losetup /dev/loop0 disk + script: + # compile v1 and v2 + - make -C v1 + - make -C v2 - - ls mount - - mkdir mount/littlefs - - cp -r $(git ls-tree --name-only HEAD) mount/littlefs - - cd mount/littlefs - - stat . - - ls -flh - - make -B test_dirs test_files QUIET=1 + # run self-host test with v1 + - v1/lfs --format /dev/loop0 + - v1/lfs /dev/loop0 mount - # attempt to migrate - - cd ../.. - - fusermount -u mount + - ls mount + - mkdir mount/littlefs + - cp -r $(git ls-tree --name-only HEAD) mount/littlefs + - cd mount/littlefs + - stat . + - ls -flh + - make -B test_dirs test_files QUIET=1 - - v2/lfs --migrate /dev/loop0 - - v2/lfs /dev/loop0 mount + # attempt to migrate + - cd ../.. + - fusermount -u mount - # run self-host test with v2 right where we left off - - ls mount - - cd mount/littlefs - - stat . - - ls -flh - - make -B test_dirs test_files QUIET=1 + - v2/lfs --migrate /dev/loop0 + - v2/lfs /dev/loop0 mount - # Automatically create releases - - stage: deploy - env: - - STAGE=deploy - - NAME=deploy - script: - - | - bash << 'SCRIPT' - set -ev - # Find version defined in lfs.h - LFS_VERSION=$(grep -ox '#define LFS_VERSION .*' lfs.h | cut -d ' ' -f3) - LFS_VERSION_MAJOR=$((0xffff & ($LFS_VERSION >> 16))) - LFS_VERSION_MINOR=$((0xffff & ($LFS_VERSION >> 0))) - # Grab latests patch from repo tags, default to 0, needs finagling - # to get past github's pagination api - PREV_URL=https://api.github.com/repos/$TRAVIS_REPO_SLUG/git/refs/tags/v$LFS_VERSION_MAJOR.$LFS_VERSION_MINOR. - PREV_URL=$(curl -u "$GEKY_BOT_RELEASES" "$PREV_URL" -I \ - | sed -n '/^Link/{s/.*<\(.*\)>; rel="last"/\1/;p;q0};$q1' \ - || echo $PREV_URL) - LFS_VERSION_PATCH=$(curl -u "$GEKY_BOT_RELEASES" "$PREV_URL" \ - | jq 'map(.ref | match("\\bv.*\\..*\\.(.*)$";"g") - .captures[].string | tonumber) | max + 1' \ - || echo 0) - # We have our new version - LFS_VERSION="v$LFS_VERSION_MAJOR.$LFS_VERSION_MINOR.$LFS_VERSION_PATCH" - echo "VERSION $LFS_VERSION" - # Check that we're the most recent commit - CURRENT_COMMIT=$(curl -f -u "$GEKY_BOT_RELEASES" \ - https://api.github.com/repos/$TRAVIS_REPO_SLUG/commits/master \ - | jq -re '.sha') - [ "$TRAVIS_COMMIT" == "$CURRENT_COMMIT" ] || exit 0 - # Create major branch - git branch v$LFS_VERSION_MAJOR HEAD - # Create major prefix branch - git config user.name "geky bot" - git config user.email "bot@geky.net" - git fetch https://github.com/$TRAVIS_REPO_SLUG.git \ - --depth=50 v$LFS_VERSION_MAJOR-prefix || true - ./scripts/prefix.py lfs$LFS_VERSION_MAJOR - git branch v$LFS_VERSION_MAJOR-prefix $( \ - git commit-tree $(git write-tree) \ - $(git rev-parse --verify -q FETCH_HEAD | sed -e 's/^/-p /') \ - -p HEAD \ - -m "Generated v$LFS_VERSION_MAJOR prefixes") - git reset --hard - # Update major version branches (vN and vN-prefix) - git push --atomic https://$GEKY_BOT_RELEASES@github.com/$TRAVIS_REPO_SLUG.git \ - v$LFS_VERSION_MAJOR \ - v$LFS_VERSION_MAJOR-prefix - # Build release notes - PREV=$(git tag --sort=-v:refname -l "v*" | head -1) - if [ ! -z "$PREV" ] - then - echo "PREV $PREV" - CHANGES=$(git log --oneline $PREV.. --grep='^Merge' --invert-grep) - printf "CHANGES\n%s\n\n" "$CHANGES" - fi - case ${GEKY_BOT_DRAFT:-minor} in - true) DRAFT=true ;; - minor) DRAFT=$(jq -R 'endswith(".0")' <<< "$LFS_VERSION") ;; - false) DRAFT=false ;; - esac - # Create the release and patch version tag (vN.N.N) - curl -f -u "$GEKY_BOT_RELEASES" -X POST \ - https://api.github.com/repos/$TRAVIS_REPO_SLUG/releases \ - -d "{ - \"tag_name\": \"$LFS_VERSION\", - \"name\": \"${LFS_VERSION%.0}\", - \"target_commitish\": \"$TRAVIS_COMMIT\", - \"draft\": $DRAFT, - \"body\": $(jq -sR '.' <<< "$CHANGES") - }" #" - SCRIPT + # run self-host test with v2 right where we left off + - ls mount + - cd mount/littlefs + - stat . + - ls -flh + - make -B test_dirs test_files QUIET=1 -# Manage statuses + # automatically create releases + - stage: deploy + env: + - NAME=deploy + script: + - | + bash << 'SCRIPT' + set -ev + # Find version defined in lfs.h + LFS_VERSION=$(grep -ox '#define LFS_VERSION .*' lfs.h | cut -d ' ' -f3) + LFS_VERSION_MAJOR=$((0xffff & ($LFS_VERSION >> 16))) + LFS_VERSION_MINOR=$((0xffff & ($LFS_VERSION >> 0))) + # Grab latests patch from repo tags, default to 0, needs finagling + # to get past github's pagination api + PREV_URL=https://api.github.com/repos/$TRAVIS_REPO_SLUG/git/refs/tags/v$LFS_VERSION_MAJOR.$LFS_VERSION_MINOR. + PREV_URL=$(curl -u "$GEKY_BOT_RELEASES" "$PREV_URL" -I \ + | sed -n '/^Link/{s/.*<\(.*\)>; rel="last"/\1/;p;q0};$q1' \ + || echo $PREV_URL) + LFS_VERSION_PATCH=$(curl -u "$GEKY_BOT_RELEASES" "$PREV_URL" \ + | jq 'map(.ref | match("\\bv.*\\..*\\.(.*)$";"g") + .captures[].string | tonumber) | max + 1' \ + || echo 0) + # We have our new version + LFS_VERSION="v$LFS_VERSION_MAJOR.$LFS_VERSION_MINOR.$LFS_VERSION_PATCH" + echo "VERSION $LFS_VERSION" + # Check that we're the most recent commit + CURRENT_COMMIT=$(curl -f -u "$GEKY_BOT_RELEASES" \ + https://api.github.com/repos/$TRAVIS_REPO_SLUG/commits/master \ + | jq -re '.sha') + [ "$TRAVIS_COMMIT" == "$CURRENT_COMMIT" ] || exit 0 + # Create major branch + git branch v$LFS_VERSION_MAJOR HEAD + # Create major prefix branch + git config user.name "geky bot" + git config user.email "bot@geky.net" + git fetch https://github.com/$TRAVIS_REPO_SLUG.git \ + --depth=50 v$LFS_VERSION_MAJOR-prefix || true + ./scripts/prefix.py lfs$LFS_VERSION_MAJOR + git branch v$LFS_VERSION_MAJOR-prefix $( \ + git commit-tree $(git write-tree) \ + $(git rev-parse --verify -q FETCH_HEAD | sed -e 's/^/-p /') \ + -p HEAD \ + -m "Generated v$LFS_VERSION_MAJOR prefixes") + git reset --hard + # Update major version branches (vN and vN-prefix) + git push --atomic https://$GEKY_BOT_RELEASES@github.com/$TRAVIS_REPO_SLUG.git \ + v$LFS_VERSION_MAJOR \ + v$LFS_VERSION_MAJOR-prefix + # Build release notes + PREV=$(git tag --sort=-v:refname -l "v*" | head -1) + if [ ! -z "$PREV" ] + then + echo "PREV $PREV" + CHANGES=$(git log --oneline $PREV.. --grep='^Merge' --invert-grep) + printf "CHANGES\n%s\n\n" "$CHANGES" + fi + case ${GEKY_BOT_DRAFT:-minor} in + true) DRAFT=true ;; + minor) DRAFT=$(jq -R 'endswith(".0")' <<< "$LFS_VERSION") ;; + false) DRAFT=false ;; + esac + # Create the release and patch version tag (vN.N.N) + curl -f -u "$GEKY_BOT_RELEASES" -X POST \ + https://api.github.com/repos/$TRAVIS_REPO_SLUG/releases \ + -d "{ + \"tag_name\": \"$LFS_VERSION\", + \"name\": \"${LFS_VERSION%.0}\", + \"target_commitish\": \"$TRAVIS_COMMIT\", + \"draft\": $DRAFT, + \"body\": $(jq -sR '.' <<< "$CHANGES") + }" #" + SCRIPT + +# manage statuses before_install: - | - curl -u "$GEKY_BOT_STATUSES" -X POST \ - https://api.github.com/repos/$TRAVIS_REPO_SLUG/statuses/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ - -d "{ - \"context\": \"$STAGE/$NAME\", - \"state\": \"pending\", - \"description\": \"${STATUS:-In progress}\", - \"target_url\": \"https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$TRAVIS_JOB_ID\" - }" - - sudo apt-get install python3-pip - - sudo pip3 install toml - + # don't clobber other (not us) failures + if ! curl https://api.github.com/repos/$TRAVIS_REPO_SLUG/status/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ + | jq -e ".statuses[] | select( + .context == \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\" and + .state == \"failure\" and + (.target_url | endswith(\"$TRAVIS_JOB_NUMBER\") | not))" + then + curl -u "$GEKY_BOT_STATUSES" -X POST \ + https://api.github.com/repos/$TRAVIS_REPO_SLUG/statuses/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ + -d "{ + \"context\": \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\", + \"state\": \"pending\", + \"description\": \"${STATUS:-In progress}\", + \"target_url\": \"$TRAVIS_JOB_WEB_URL#$TRAVIS_JOB_NUMBER\" + }" + fi after_failure: - | - curl -u "$GEKY_BOT_STATUSES" -X POST \ - https://api.github.com/repos/$TRAVIS_REPO_SLUG/statuses/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ - -d "{ - \"context\": \"$STAGE/$NAME\", - \"state\": \"failure\", - \"description\": \"${STATUS:-Failed}\", - \"target_url\": \"https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$TRAVIS_JOB_ID\" - }" + # don't clobber other (not us) failures + if ! curl https://api.github.com/repos/$TRAVIS_REPO_SLUG/status/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ + | jq -e ".statuses[] | select( + .context == \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\" and + .state == \"failure\" and + (.target_url | endswith(\"$TRAVIS_JOB_NUMBER\") | not))" + then + curl -u "$GEKY_BOT_STATUSES" -X POST \ + https://api.github.com/repos/$TRAVIS_REPO_SLUG/statuses/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ + -d "{ + \"context\": \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\", + \"state\": \"failure\", + \"description\": \"${STATUS:-Failed}\", + \"target_url\": \"$TRAVIS_JOB_WEB_URL#$TRAVIS_JOB_NUMBER\" + }" + fi after_success: - | - curl -u "$GEKY_BOT_STATUSES" -X POST \ - https://api.github.com/repos/$TRAVIS_REPO_SLUG/statuses/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ - -d "{ - \"context\": \"$STAGE/$NAME\", - \"state\": \"success\", - \"description\": \"${STATUS:-Passed}\", - \"target_url\": \"https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$TRAVIS_JOB_ID\" - }" - -# Job control -stages: - - name: test - - name: deploy - if: branch = master AND type = push + # don't clobber other (not us) failures + # only update if we were last job to mark in progress, + # this isn't perfect but is probably good enough + if ! curl https://api.github.com/repos/$TRAVIS_REPO_SLUG/status/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ + | jq -e ".statuses[] | select( + .context == \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\" and + (.state == \"failure\" or .state == \"pending\") and + (.target_url | endswith(\"$TRAVIS_JOB_NUMBER\") | not))" + then + curl -u "$GEKY_BOT_STATUSES" -X POST \ + https://api.github.com/repos/$TRAVIS_REPO_SLUG/statuses/${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT} \ + -d "{ + \"context\": \"${TRAVIS_BUILD_STAGE_NAME,,}/$NAME\", + \"state\": \"success\", + \"description\": \"${STATUS:-Passed}\", + \"target_url\": \"$TRAVIS_JOB_WEB_URL#$TRAVIS_JOB_NUMBER\" + }" + fi From d04b07750619f1a21f0ecf8e8e3c7cd21ff2ebb4 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Mon, 17 Feb 2020 12:00:47 -0600 Subject: [PATCH 29/41] Fixed minor things to get CI passing again - Added caching to Travis install dirs, because otherwise pip3 install fails randomly - Increased size of littlefs-fuse disk because test script has a larger footprint now - Skip a couple of reentrant tests under byte-level writes because the tests just take too long and cause Travis to bail due to no output for 10m - Fixed various Valgrind errors - Suppressed uninit checks for tests where LFS_BLOCK_ERASE_VALUE == -1. In this case rambd goes uninitialized, which is fine for rambd's purposes. Note I couldn't figure out how to limit this suppression to only the malloc in rambd, this doesn't seem possible with Valgrind. - Fixed memory leaks in exhaustion tests - Fixed off-by-1 string null-terminator issue in paths tests - Fixed lfs_file_sync issue caused by revealed by fixing memory leaks in exhaustion tests. Getting ENOSPC during a file write puts the file in a bad state where littlefs doesn't know how to write it out safely. In this case, lfs_file_sync and lfs_file_close return 0 without writing out state so that device-side resources can still be cleaned up. To recover from ENOSPC, the file needs to be reopened and the writes recreated. Not sure if there is a better way to handle this. - Added some quality-of-life improvements to Valgrind testing - Fit Valgrind messages into truncated output when not in verbose mode - Turned on origin tracking --- .travis.yml | 32 ++++++++++++++------------------ lfs.c | 7 ++++++- scripts/test.py | 6 ++++++ tests/test_dirs.toml | 2 ++ tests/test_exhaustion.toml | 20 ++++++++++++++++++++ tests/test_paths.toml | 5 ++--- 6 files changed, 50 insertions(+), 22 deletions(-) diff --git a/.travis.yml b/.travis.yml index 5d10d4ee..0ee90a27 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,10 +4,16 @@ env: - CFLAGS=-Werror - MAKEFLAGS=-j +# cache installation dirs +cache: + pip: true + directories: + - $HOME/.cache/apt + # common installation _: &install-common # need toml, also pip3 isn't installed by default? - - sudo apt-get install python3-pip + - sudo apt-get install python3 python3-pip - sudo pip3 install toml # setup a ram-backed disk to speed up reentrant tests - mkdir disks @@ -189,22 +195,12 @@ jobs: stage: test env: - NAME=littlefs-valgrind - - TFLAGS="$TFLAGS --valgrind" install: - *install-common - sudo apt-get install valgrind - valgrind --version - script: *test-example - - {<<: *valgrind, script: *test-default} - - {<<: *valgrind, script: *test-nor} - - {<<: *valgrind, script: *test-emmc} - - {<<: *valgrind, script: *test-nand} - - {<<: *valgrind, script: *test-no-intrinsics} - - {<<: *valgrind, script: *test-no-inline} - - {<<: *valgrind, script: *test-byte-writes} - - {<<: *valgrind, script: *test-block-cycles} - - {<<: *valgrind, script: *test-odd-block-count} - - {<<: *valgrind, script: *test-odd-block-size} + script: + - make test TFLAGS+="-k --valgrind" # self-host with littlefs-fuse for fuzz test - stage: test @@ -224,7 +220,7 @@ jobs: - mkdir mount - sudo chmod a+rw /dev/loop0 - - dd if=/dev/zero bs=512 count=4096 of=disk + - dd if=/dev/zero bs=512 count=128K of=disk - losetup /dev/loop0 disk script: # self-host test @@ -239,7 +235,7 @@ jobs: - cd mount/littlefs - stat . - ls -flh - - make -B test_dirs test_files QUIET=1 + - make -B test # test migration using littlefs-fuse - stage: test @@ -260,7 +256,7 @@ jobs: - mkdir mount - sudo chmod a+rw /dev/loop0 - - dd if=/dev/zero bs=512 count=4096 of=disk + - dd if=/dev/zero bs=512 count=128K of=disk - losetup /dev/loop0 disk script: # compile v1 and v2 @@ -277,7 +273,7 @@ jobs: - cd mount/littlefs - stat . - ls -flh - - make -B test_dirs test_files QUIET=1 + - make -B test # attempt to migrate - cd ../.. @@ -291,7 +287,7 @@ jobs: - cd mount/littlefs - stat . - ls -flh - - make -B test_dirs test_files QUIET=1 + - make -B test # automatically create releases - stage: deploy diff --git a/lfs.c b/lfs.c index 354cebc7..71903121 100644 --- a/lfs.c +++ b/lfs.c @@ -2707,6 +2707,12 @@ int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) { LFS_TRACE("lfs_file_sync(%p, %p)", (void*)lfs, (void*)file); LFS_ASSERT(file->flags & LFS_F_OPENED); + if (file->flags & LFS_F_ERRED) { + // it's not safe to do anything if our file errored + LFS_TRACE("lfs_file_sync -> %d", 0); + return 0; + } + int err = lfs_file_flush(lfs, file); if (err) { file->flags |= LFS_F_ERRED; @@ -2715,7 +2721,6 @@ int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) { } if ((file->flags & LFS_F_DIRTY) && - !(file->flags & LFS_F_ERRED) && !lfs_pair_isnull(file->m.pair)) { // update dir entry uint16_t type; diff --git a/scripts/test.py b/scripts/test.py index 018d0b23..61870289 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -299,10 +299,16 @@ def shouldtest(self, **args): return not self.leaky and super().shouldtest(**args) def test(self, exec=[], **args): + verbose = args.get('verbose', False) + uninit = (self.defines.get('LFS_ERASE_VALUE', None) == -1) exec = [ 'valgrind', '--leak-check=full', + ] + (['--undef-value-errors=no'] if uninit else []) + [ + ] + (['--track-origins=yes'] if not uninit else []) + [ '--error-exitcode=4', + '--error-limit=no', + ] + (['--num-callers=1'] if not verbose else []) + [ '-q'] + exec return super().test(exec=exec, **args) diff --git a/tests/test_dirs.toml b/tests/test_dirs.toml index 270f4f8e..6cfd7902 100644 --- a/tests/test_dirs.toml +++ b/tests/test_dirs.toml @@ -157,6 +157,7 @@ code = ''' [[case]] # reentrant many directory creation/rename/removal define.N = [5, 11] reentrant = true +if = 'LFS_CACHE_SIZE >= 4' # these just take too long with byte-level writes code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -383,6 +384,7 @@ code = ''' [[case]] # reentrant file creation/rename/removal define.N = [5, 25] reentrant = true +if = 'LFS_CACHE_SIZE >= 4' # these just take too long with byte-level writes code = ''' err = lfs_mount(&lfs, &cfg); if (err) { diff --git a/tests/test_exhaustion.toml b/tests/test_exhaustion.toml index 26eeeba2..569611c5 100644 --- a/tests/test_exhaustion.toml +++ b/tests/test_exhaustion.toml @@ -33,6 +33,9 @@ code = ''' lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); assert(res == 1 || res == LFS_ERR_NOSPC); if (res == LFS_ERR_NOSPC) { + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -40,6 +43,7 @@ code = ''' err = lfs_file_close(&lfs, &file); assert(err == 0 || err == LFS_ERR_NOSPC); if (err == LFS_ERR_NOSPC) { + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -111,6 +115,9 @@ code = ''' lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); assert(res == 1 || res == LFS_ERR_NOSPC); if (res == LFS_ERR_NOSPC) { + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -118,6 +125,7 @@ code = ''' err = lfs_file_close(&lfs, &file); assert(err == 0 || err == LFS_ERR_NOSPC); if (err == LFS_ERR_NOSPC) { + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -198,6 +206,9 @@ code = ''' lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); assert(res == 1 || res == LFS_ERR_NOSPC); if (res == LFS_ERR_NOSPC) { + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -205,6 +216,7 @@ code = ''' err = lfs_file_close(&lfs, &file); assert(err == 0 || err == LFS_ERR_NOSPC); if (err == LFS_ERR_NOSPC) { + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -283,6 +295,9 @@ code = ''' lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); assert(res == 1 || res == LFS_ERR_NOSPC); if (res == LFS_ERR_NOSPC) { + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -290,6 +305,7 @@ code = ''' err = lfs_file_close(&lfs, &file); assert(err == 0 || err == LFS_ERR_NOSPC); if (err == LFS_ERR_NOSPC) { + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -364,6 +380,9 @@ code = ''' lfs_ssize_t res = lfs_file_write(&lfs, &file, &c, 1); assert(res == 1 || res == LFS_ERR_NOSPC); if (res == LFS_ERR_NOSPC) { + err = lfs_file_close(&lfs, &file); + assert(err == 0 || err == LFS_ERR_NOSPC); + lfs_unmount(&lfs) => 0; goto exhausted; } } @@ -371,6 +390,7 @@ code = ''' err = lfs_file_close(&lfs, &file); assert(err == 0 || err == LFS_ERR_NOSPC); if (err == LFS_ERR_NOSPC) { + lfs_unmount(&lfs) => 0; goto exhausted; } } diff --git a/tests/test_paths.toml b/tests/test_paths.toml index 480dd9ac..a7474c0b 100644 --- a/tests/test_paths.toml +++ b/tests/test_paths.toml @@ -247,14 +247,14 @@ code = ''' lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; memset(path, 'w', LFS_NAME_MAX+1); - path[LFS_NAME_MAX+2] = '\0'; + path[LFS_NAME_MAX+1] = '\0'; lfs_mkdir(&lfs, path) => LFS_ERR_NAMETOOLONG; lfs_file_open(&lfs, &file, path, LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_NAMETOOLONG; memcpy(path, "coffee/", strlen("coffee/")); memset(path+strlen("coffee/"), 'w', LFS_NAME_MAX+1); - path[strlen("coffee/")+LFS_NAME_MAX+2] = '\0'; + path[strlen("coffee/")+LFS_NAME_MAX+1] = '\0'; lfs_mkdir(&lfs, path) => LFS_ERR_NAMETOOLONG; lfs_file_open(&lfs, &file, path, LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_NAMETOOLONG; @@ -270,7 +270,6 @@ code = ''' lfs_mkdir(&lfs, "coffee/warmcoffee") => 0; lfs_mkdir(&lfs, "coffee/coldcoffee") => 0; - lfs_mount(&lfs, &cfg) => 0; memset(path, 'w', LFS_NAME_MAX); path[LFS_NAME_MAX] = '\0'; lfs_mkdir(&lfs, path) => 0; From 09902966199cc73977767daa36aa786d2b2f8d0c Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 18 Feb 2020 11:35:22 -0600 Subject: [PATCH 30/41] Limited byte-level tests to native testing due to time Byte-level writes are expensive and not suggested (caches >= 4 bytes make much more sense), however there are many corner cases with byte-level writes that can be easy to miss (power-loss leaving single bytes written to disk). Unfortunately, byte-level writes mixed with power-loss testing, the Travis infrastructure, and Arm Thumb instruction set simulation exceeds the 50-minute budget Travis allocates for jobs. For now I'm disabling the byte-level tests under Qemu, with the hope that performance improvements in littlefs will let us turn these tests back on in the future. --- .travis.yml | 18 ++++++++++++------ tests/test_dirs.toml | 2 -- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/.travis.yml b/.travis.yml index 0ee90a27..78d964a4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -115,7 +115,7 @@ jobs: env: - NAME=littlefs-arm - CC="arm-linux-gnueabi-gcc --static -mthumb" - - TFLAGS="$TFLAGS -e qemu-arm" + - TFLAGS="$TFLAGS --exec=qemu-arm" install: - *install-common - sudo apt-get install @@ -131,7 +131,9 @@ jobs: - {<<: *arm, script: [*test-nand, *report-size]} - {<<: *arm, script: [*test-no-intrinsics, *report-size]} - {<<: *arm, script: [*test-no-inline, *report-size]} - - {<<: *arm, script: [*test-byte-writes, *report-size]} + # it just takes way to long to run byte-level writes in qemu, + # note this is still tested in the native tests + #- {<<: *arm, script: [*test-byte-writes, *report-size]} - {<<: *arm, script: [*test-block-cycles, *report-size]} - {<<: *arm, script: [*test-odd-block-count, *report-size]} - {<<: *arm, script: [*test-odd-block-size, *report-size]} @@ -142,7 +144,7 @@ jobs: env: - NAME=littlefs-mips - CC="mips-linux-gnu-gcc --static" - - TFLAGS="$TFLAGS -e qemu-mips" + - TFLAGS="$TFLAGS --exec=qemu-mips" install: - *install-common - sudo apt-get install @@ -158,7 +160,9 @@ jobs: - {<<: *mips, script: [*test-nand, *report-size]} - {<<: *mips, script: [*test-no-intrinsics, *report-size]} - {<<: *mips, script: [*test-no-inline, *report-size]} - - {<<: *mips, script: [*test-byte-writes, *report-size]} + # it just takes way to long to run byte-level writes in qemu, + # note this is still tested in the native tests + #- {<<: *mips, script: [*test-byte-writes, *report-size]} - {<<: *mips, script: [*test-block-cycles, *report-size]} - {<<: *mips, script: [*test-odd-block-count, *report-size]} - {<<: *mips, script: [*test-odd-block-size, *report-size]} @@ -169,7 +173,7 @@ jobs: env: - NAME=littlefs-powerpc - CC="powerpc-linux-gnu-gcc --static" - - TFLAGS="$TFLAGS -e qemu-ppc" + - TFLAGS="$TFLAGS --exec=qemu-ppc" install: - *install-common - sudo apt-get install @@ -185,7 +189,9 @@ jobs: - {<<: *powerpc, script: [*test-nand, *report-size]} - {<<: *powerpc, script: [*test-no-intrinsics, *report-size]} - {<<: *powerpc, script: [*test-no-inline, *report-size]} - - {<<: *powerpc, script: [*test-byte-writes, *report-size]} + # it just takes way to long to run byte-level writes in qemu, + # note this is still tested in the native tests + #- {<<: *powerpc, script: [*test-byte-writes, *report-size]} - {<<: *powerpc, script: [*test-block-cycles, *report-size]} - {<<: *powerpc, script: [*test-odd-block-count, *report-size]} - {<<: *powerpc, script: [*test-odd-block-size, *report-size]} diff --git a/tests/test_dirs.toml b/tests/test_dirs.toml index 6cfd7902..270f4f8e 100644 --- a/tests/test_dirs.toml +++ b/tests/test_dirs.toml @@ -157,7 +157,6 @@ code = ''' [[case]] # reentrant many directory creation/rename/removal define.N = [5, 11] reentrant = true -if = 'LFS_CACHE_SIZE >= 4' # these just take too long with byte-level writes code = ''' err = lfs_mount(&lfs, &cfg); if (err) { @@ -384,7 +383,6 @@ code = ''' [[case]] # reentrant file creation/rename/removal define.N = [5, 25] reentrant = true -if = 'LFS_CACHE_SIZE >= 4' # these just take too long with byte-level writes code = ''' err = lfs_mount(&lfs, &cfg); if (err) { From 50fe8ae258b8e80c04a3c64e6d91885287c49aa5 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 18 Feb 2020 18:08:24 -0600 Subject: [PATCH 31/41] Renamed test_format -> test_superblocks, tweaked superblock tests With the superblock expansion stuff, the test_format tests have grown to test more advanced superblock-related features. This is fine but deserves a rename so it's more clear. Also fixed a typo that meant tests never ran with block cycles. --- ...test_format.toml => test_superblocks.toml} | 46 ++++++++++++++----- 1 file changed, 35 insertions(+), 11 deletions(-) rename tests/{test_format.toml => test_superblocks.toml} (62%) diff --git a/tests/test_format.toml b/tests/test_superblocks.toml similarity index 62% rename from tests/test_format.toml rename to tests/test_superblocks.toml index 7932d54b..407c8454 100644 --- a/tests/test_format.toml +++ b/tests/test_superblocks.toml @@ -27,41 +27,55 @@ code = ''' ''' [[case]] # expanding superblock -define.BLOCK_CYCLES = [32, 33, 1] +define.LFS_BLOCK_CYCLES = [32, 33, 1] define.N = [10, 100, 1000] code = ''' lfs_format(&lfs, &cfg) => 0; lfs_mount(&lfs, &cfg) => 0; for (int i = 0; i < N; i++) { - lfs_mkdir(&lfs, "dummy") => 0; + lfs_file_open(&lfs, &file, "dummy", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + lfs_file_close(&lfs, &file) => 0; lfs_stat(&lfs, "dummy", &info) => 0; assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); lfs_remove(&lfs, "dummy") => 0; } lfs_unmount(&lfs) => 0; // one last check after power-cycle lfs_mount(&lfs, &cfg) => 0; - lfs_mkdir(&lfs, "dummy") => 0; + lfs_file_open(&lfs, &file, "dummy", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + lfs_file_close(&lfs, &file) => 0; lfs_stat(&lfs, "dummy", &info) => 0; assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); lfs_unmount(&lfs) => 0; ''' [[case]] # expanding superblock with power cycle -define.BLOCK_CYCLES = [32, 33, 1] +define.LFS_BLOCK_CYCLES = [32, 33, 1] define.N = [10, 100, 1000] code = ''' lfs_format(&lfs, &cfg) => 0; for (int i = 0; i < N; i++) { lfs_mount(&lfs, &cfg) => 0; // remove lingering dummy? - err = lfs_remove(&lfs, "dummy"); + err = lfs_stat(&lfs, "dummy", &info); assert(err == 0 || (err == LFS_ERR_NOENT && i == 0)); - - lfs_mkdir(&lfs, "dummy") => 0; + if (!err) { + assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); + lfs_remove(&lfs, "dummy") => 0; + } + + lfs_file_open(&lfs, &file, "dummy", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + lfs_file_close(&lfs, &file) => 0; lfs_stat(&lfs, "dummy", &info) => 0; assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); lfs_unmount(&lfs) => 0; } @@ -69,11 +83,12 @@ code = ''' lfs_mount(&lfs, &cfg) => 0; lfs_stat(&lfs, "dummy", &info) => 0; assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); lfs_unmount(&lfs) => 0; ''' [[case]] # reentrant expanding superblock -define.BLOCK_CYCLES = [2, 1] +define.LFS_BLOCK_CYCLES = [2, 1] define.N = 24 reentrant = true code = ''' @@ -85,12 +100,20 @@ code = ''' for (int i = 0; i < N; i++) { // remove lingering dummy? - err = lfs_remove(&lfs, "dummy"); + err = lfs_stat(&lfs, "dummy", &info); assert(err == 0 || (err == LFS_ERR_NOENT && i == 0)); - - lfs_mkdir(&lfs, "dummy") => 0; + if (!err) { + assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); + lfs_remove(&lfs, "dummy") => 0; + } + + lfs_file_open(&lfs, &file, "dummy", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + lfs_file_close(&lfs, &file) => 0; lfs_stat(&lfs, "dummy", &info) => 0; assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); } lfs_unmount(&lfs) => 0; @@ -99,5 +122,6 @@ code = ''' lfs_mount(&lfs, &cfg) => 0; lfs_stat(&lfs, "dummy", &info) => 0; assert(strcmp(info.name, "dummy") == 0); + assert(info.type == LFS_TYPE_REG); lfs_unmount(&lfs) => 0; ''' From a7dfae4526bb77ec7db8a29072ed381e1b19c6f1 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sat, 22 Feb 2020 23:36:45 -0600 Subject: [PATCH 32/41] Minor tweaks to debugging scripts, fixed explode_asserts.py off-by-1 - Changed readmdir.py to print the metadata pair and revision count, which is useful when debugging commit issues. - Added truncated data view to readtree.py by default. This does mean readtree.py must read all files on the filesystem to show the truncated data, hopefully this does not end up being a problem. - Made overall representation hopefully more readable, including moving superblock under the root dir, userattrs under files, fixing a gstate rendering issue. - Added rendering of soft-tails as dotted-arrows, hopefully this isn't too noisy. - Fixed explode_asserts.py off-by-1 in #line mapping caused by a strip call in the assert generation eating newlines. The script matches line numbers between the original+modified files by emitting assert statements that use the same number of lines. An off-by-1 here causes the entire file to map lines incorrectly, which can be very annoying. --- scripts/explode_asserts.py | 4 +- scripts/readmdir.py | 8 ++ scripts/readtree.py | 168 ++++++++++++++++++------------------- 3 files changed, 94 insertions(+), 86 deletions(-) diff --git a/scripts/explode_asserts.py b/scripts/explode_asserts.py index c0534cb1..8a8e5b1c 100755 --- a/scripts/explode_asserts.py +++ b/scripts/explode_asserts.py @@ -166,8 +166,8 @@ def mkassert(type, comp, lh, rh, size=None): 'type': type.lower(), 'TYPE': type.upper(), 'comp': comp.lower(), 'COMP': comp.upper(), 'prefix': PREFIX.lower(), 'PREFIX': PREFIX.upper(), - 'lh': lh.strip(), - 'rh': rh.strip(), + 'lh': lh.strip(' '), + 'rh': rh.strip(' '), 'size': size, } if size: diff --git a/scripts/readmdir.py b/scripts/readmdir.py index ef634a96..75f8b9ac 100755 --- a/scripts/readmdir.py +++ b/scripts/readmdir.py @@ -318,6 +318,14 @@ def main(args): # find most recent pair mdir = MetadataPair(blocks) + print("mdir {%s} rev %d%s%s" % ( + ', '.join('%#x' % b + for b in [args.block1, args.block2] + if b is not None), + mdir.rev, + ' (was %s)' % ', '.join('%d' % m.rev for m in mdir.pair[1:]) + if len(mdir.pair) > 1 else '', + ' (corrupted)' if not mdir else '')) if args.all: mdir.dump_all(truncate=not args.no_truncate) elif args.log: diff --git a/scripts/readtree.py b/scripts/readtree.py index ea8cb5f5..3965a36c 100755 --- a/scripts/readtree.py +++ b/scripts/readtree.py @@ -18,26 +18,22 @@ def dumpentries(args, mdir, f): name = mdir[Tag('name', id_, 0)] struct_ = mdir[Tag('struct', id_, 0)] - f.write("id %d %s %s" % ( + desc = "id %d %s %s" % ( id_, name.typerepr(), - json.dumps(name.data.decode('utf8')))) + json.dumps(name.data.decode('utf8'))) if struct_.is_('dirstruct'): - f.write(" dir {%#x, %#x}" % struct.unpack( - '= ' ' and c <= '~' else '.' - for c in map(chr, struct_.data[i:i+16])))) - elif args.data and struct_.is_('ctzstruct'): + data = None + if struct_.is_('inlinestruct'): + data = struct_.data + elif struct_.is_('ctzstruct'): block, size = struct.unpack( '= ' ' and c <= '~' else '.' - for c in map(chr, data[i:i+16])))) + + f.write("%-45s%s\n" % (desc, + "%-23s %-8s" % ( + ' '.join('%02x' % c for c in data[:8]), + ''.join(c if c >= ' ' and c <= '~' else '.' + for c in map(chr, data[:8]))) + if not args.no_truncate and len(desc) < 45 + and data is not None else "")) + + if name.is_('superblock') and struct_.is_('inlinestruct'): + f.write( + " block_size %d\n" + " block_count %d\n" + " name_max %d\n" + " file_max %d\n" + " attr_max %d\n" % struct.unpack( + '= ' ' and c <= '~' else '.' + for c in map(chr, tag.data[:8]))) + if not args.no_truncate and len(desc) < 43 else "")) - if args.data: + if args.no_truncate: for i in range(0, len(tag.data), 16): - f.write(" %-47s %-16s\n" % ( - ' '.join('%02x' % c for c in tag.data[i:i+16]), + f.write(" %08x: %-47s %-16s\n" % ( + i, ' '.join('%02x' % c for c in tag.data[i:i+16]), ''.join(c if c >= ' ' and c <= '~' else '.' for c in map(chr, tag.data[i:i+16])))) + if args.no_truncate and data is not None: + for i in range(0, len(data), 16): + f.write(" %08x: %-47s %-16s\n" % ( + i, ' '.join('%02x' % c for c in data[i:i+16]), + ''.join(c if c >= ' ' and c <= '~' else '.' + for c in map(chr, data[i:i+16])))) + def main(args): with open(args.disk, 'rb') as f: dirs = [] @@ -161,61 +179,51 @@ def main(args): dir[0].path = path.replace('//', '/') # dump tree - if not args.superblock and not args.gstate and not args.mdirs: - args.superblock = True - args.gstate = True - args.mdirs = True - - if args.superblock and superblock: - print("superblock %s v%d.%d" % ( - json.dumps(superblock[0].data.decode('utf8')), - struct.unpack('=%d" % max(tag.size, 1)) if tag.type: print(" move dir {%#x, %#x} id %d" % ( blocks[0], blocks[1], tag.id)) - if args.mdirs: - for i, dir in enumerate(dirs): - print("dir %s" % (json.dumps(dir[0].path) - if hasattr(dir[0], 'path') else '(orphan)')) - - for j, mdir in enumerate(dir): - print("mdir {%#x, %#x} rev %d%s" % ( - mdir.blocks[0], mdir.blocks[1], mdir.rev, - ' (corrupted)' if not mdir else '')) - - f = io.StringIO() - if args.tags: - mdir.dump_tags(f, truncate=not args.no_truncate) - elif args.log: - mdir.dump_log(f, truncate=not args.no_truncate) - elif args.all: - mdir.dump_all(f, truncate=not args.no_truncate) - else: - dumpentries(args, mdir, f) - - lines = list(filter(None, f.getvalue().split('\n'))) - for k, line in enumerate(lines): - print("%s %s" % ( - ' ' if j == len(dir)-1 else - 'v' if k == len(lines)-1 else - '|', - line)) + for i, dir in enumerate(dirs): + print("dir %s" % (json.dumps(dir[0].path) + if hasattr(dir[0], 'path') else '(orphan)')) + + for j, mdir in enumerate(dir): + print("mdir {%#x, %#x} rev %d%s" % ( + mdir.blocks[0], mdir.blocks[1], mdir.rev, + ' (corrupted)' if not mdir else '')) + + f = io.StringIO() + if args.tags: + mdir.dump_tags(f, truncate=not args.no_truncate) + elif args.log: + mdir.dump_log(f, truncate=not args.no_truncate) + elif args.all: + mdir.dump_all(f, truncate=not args.no_truncate) + else: + dumpentries(args, mdir, f) + + lines = list(filter(None, f.getvalue().split('\n'))) + for k, line in enumerate(lines): + print("%s %s" % ( + ' ' if i == len(dirs)-1 and j == len(dir)-1 else + 'v' if k == len(lines)-1 else + '.' if j == len(dir)-1 else + '|', + line)) if cycle: print("*** cycle detected! -> {%#x, %#x} ***" % (cycle[0], cycle[1])) @@ -242,20 +250,12 @@ def main(args): parser.add_argument('block2', nargs='?', default=1, type=lambda x: int(x, 0), help="Optional second block address for finding the root.") - parser.add_argument('-s', '--superblock', action='store_true', - help="Show contents of the superblock.") - parser.add_argument('-g', '--gstate', action='store_true', - help="Show contents of global-state.") - parser.add_argument('-m', '--mdirs', action='store_true', - help="Show contents of metadata-pairs/directories.") parser.add_argument('-t', '--tags', action='store_true', help="Show metadata tags instead of reconstructing entries.") parser.add_argument('-l', '--log', action='store_true', help="Show tags in log.") parser.add_argument('-a', '--all', action='store_true', help="Show all tags in log, included tags in corrupted commits.") - parser.add_argument('-d', '--data', action='store_true', - help="Also show the raw contents of files/attrs/tags.") parser.add_argument('-T', '--no-truncate', action='store_true', - help="Don't truncate large amounts of data.") + help="Show the full contents of files/attrs/tags.") sys.exit(main(parser.parse_args())) From cb26157880e475adc97104ef2954707bc8ec5628 Mon Sep 17 00:00:00 2001 From: Chris Desjardins Date: Mon, 30 Dec 2019 12:56:27 +0100 Subject: [PATCH 33/41] Change assert to runtime check. I had a system that was constantly hitting this assert, after making this change it recovered immediately. --- lfs.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lfs.c b/lfs.c index 71903121..0bb570cb 100644 --- a/lfs.c +++ b/lfs.c @@ -29,8 +29,7 @@ static int lfs_bd_read(lfs_t *lfs, lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size) { uint8_t *data = buffer; - LFS_ASSERT(block != LFS_BLOCK_NULL); - if (off+size > lfs->cfg->block_size) { + if ((off+size > lfs->cfg->block_size) || (block == LFS_BLOCK_NULL)) { return LFS_ERR_CORRUPT; } From 4677421ababf7be568d27167a51a211749329400 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 23 Feb 2020 22:55:52 -0600 Subject: [PATCH 34/41] Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem. --- lfs.c | 44 ++++++- tests/test_evil.toml | 288 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 326 insertions(+), 6 deletions(-) create mode 100644 tests/test_evil.toml diff --git a/lfs.c b/lfs.c index 0bb570cb..93a124d4 100644 --- a/lfs.c +++ b/lfs.c @@ -29,7 +29,8 @@ static int lfs_bd_read(lfs_t *lfs, lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size) { uint8_t *data = buffer; - if ((off+size > lfs->cfg->block_size) || (block == LFS_BLOCK_NULL)) { + if (block >= lfs->cfg->block_count || + off+size > lfs->cfg->block_size) { return LFS_ERR_CORRUPT; } @@ -172,7 +173,7 @@ static int lfs_bd_prog(lfs_t *lfs, lfs_block_t block, lfs_off_t off, const void *buffer, lfs_size_t size) { const uint8_t *data = buffer; - LFS_ASSERT(block != LFS_BLOCK_NULL); + LFS_ASSERT(block == LFS_BLOCK_INLINE || block < lfs->cfg->block_count); LFS_ASSERT(off + size <= lfs->cfg->block_size); while (size > 0) { @@ -747,6 +748,12 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, // scanning the entire directory lfs_stag_t besttag = -1; + // if either block address is invalid we return LFS_ERR_CORRUPT here, + // otherwise later writes to the pair could fail + if (pair[0] >= lfs->cfg->block_count || pair[1] >= lfs->cfg->block_count) { + return LFS_ERR_CORRUPT; + } + // find the block with the most recent revision uint32_t revs[2] = {0, 0}; int r = 0; @@ -2196,7 +2203,6 @@ static int lfs_ctz_find(lfs_t *lfs, return err; } - LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count); current -= 1 << skip; } @@ -2216,7 +2222,6 @@ static int lfs_ctz_extend(lfs_t *lfs, if (err) { return err; } - LFS_ASSERT(nblock >= 2 && nblock <= lfs->cfg->block_count); { err = lfs_bd_erase(lfs, nblock); @@ -2289,8 +2294,6 @@ static int lfs_ctz_extend(lfs_t *lfs, return err; } } - - LFS_ASSERT(nhead >= 2 && nhead <= lfs->cfg->block_count); } *block = nblock; @@ -3661,7 +3664,15 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { // scan directory blocks for superblock and any global updates lfs_mdir_t dir = {.tail = {0, 1}}; + lfs_block_t cycle = 0; while (!lfs_pair_isnull(dir.tail)) { + if (cycle >= lfs->cfg->block_count/2) { + // loop detected + err = LFS_ERR_CORRUPT; + goto cleanup; + } + cycle += 1; + // fetch next block in tail list lfs_stag_t tag = lfs_dir_fetchmatch(lfs, &dir, dir.tail, LFS_MKTAG(0x7ff, 0x3ff, 0), @@ -3803,7 +3814,14 @@ int lfs_fs_traverseraw(lfs_t *lfs, } #endif + lfs_block_t cycle = 0; while (!lfs_pair_isnull(dir.tail)) { + if (cycle >= lfs->cfg->block_count/2) { + // loop detected + return LFS_ERR_CORRUPT; + } + cycle += 1; + for (int i = 0; i < 2; i++) { int err = cb(data, dir.tail[i]); if (err) { @@ -3887,7 +3905,14 @@ static int lfs_fs_pred(lfs_t *lfs, // iterate over all directory directory entries pdir->tail[0] = 0; pdir->tail[1] = 1; + lfs_block_t cycle = 0; while (!lfs_pair_isnull(pdir->tail)) { + if (cycle >= lfs->cfg->block_count/2) { + // loop detected + return LFS_ERR_CORRUPT; + } + cycle += 1; + if (lfs_pair_cmp(pdir->tail, pair) == 0) { return 0; } @@ -3930,7 +3955,14 @@ static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2], // use fetchmatch with callback to find pairs parent->tail[0] = 0; parent->tail[1] = 1; + lfs_block_t cycle = 0; while (!lfs_pair_isnull(parent->tail)) { + if (cycle >= lfs->cfg->block_count/2) { + // loop detected + return LFS_ERR_CORRUPT; + } + cycle += 1; + lfs_stag_t tag = lfs_dir_fetchmatch(lfs, parent, parent->tail, LFS_MKTAG(0x7ff, 0, 0x3ff), LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, 8), diff --git a/tests/test_evil.toml b/tests/test_evil.toml new file mode 100644 index 00000000..920d3a0e --- /dev/null +++ b/tests/test_evil.toml @@ -0,0 +1,288 @@ +# Tests for recovering from conditions which shouldn't normally +# happen during normal operation of littlefs + +# invalid pointer tests (outside of block_count) + +[[case]] # invalid tail-pointer test +define.TAIL_TYPE = ['LFS_TYPE_HARDTAIL', 'LFS_TYPE_SOFTTAIL'] +define.INVALSET = [0x3, 0x1, 0x2] +in = "lfs.c" +code = ''' + // create littlefs + lfs_format(&lfs, &cfg) => 0; + + // change tail-pointer to invalid pointers + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + lfs_dir_commit(&lfs, &mdir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_HARDTAIL, 0x3ff, 8), + (lfs_block_t[2]){ + (INVALSET & 0x1) ? 0xcccccccc : 0, + (INVALSET & 0x2) ? 0xcccccccc : 0}})) => 0; + lfs_deinit(&lfs) => 0; + + // test that mount fails gracefully + lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; +''' + +[[case]] # invalid dir pointer test +define.INVALSET = [0x3, 0x1, 0x2] +in = "lfs.c" +code = ''' + // create littlefs + lfs_format(&lfs, &cfg) => 0; + // make a dir + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "dir_here") => 0; + lfs_unmount(&lfs) => 0; + + // change the dir pointer to be invalid + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + // make sure id 1 == our directory + lfs_dir_get(&lfs, &mdir, + LFS_MKTAG(0x700, 0x3ff, 0), + LFS_MKTAG(LFS_TYPE_NAME, 1, strlen("dir_here")), buffer) + => LFS_MKTAG(LFS_TYPE_DIR, 1, strlen("dir_here")); + assert(memcmp((char*)buffer, "dir_here", strlen("dir_here")) == 0); + // change dir pointer + lfs_dir_commit(&lfs, &mdir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 1, 8), + (lfs_block_t[2]){ + (INVALSET & 0x1) ? 0xcccccccc : 0, + (INVALSET & 0x2) ? 0xcccccccc : 0}})) => 0; + lfs_deinit(&lfs) => 0; + + // test that accessing our bad dir fails, note there's a number + // of ways to access the dir, some can fail, but some don't + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "dir_here", &info) => 0; + assert(strcmp(info.name, "dir_here") == 0); + assert(info.type == LFS_TYPE_DIR); + + lfs_dir_open(&lfs, &dir, "dir_here") => LFS_ERR_CORRUPT; + lfs_stat(&lfs, "dir_here/file_here", &info) => LFS_ERR_CORRUPT; + lfs_dir_open(&lfs, &dir, "dir_here/dir_here") => LFS_ERR_CORRUPT; + lfs_file_open(&lfs, &file, "dir_here/file_here", + LFS_O_RDONLY) => LFS_ERR_CORRUPT; + lfs_file_open(&lfs, &file, "dir_here/file_here", + LFS_O_WRONLY | LFS_O_CREAT) => LFS_ERR_CORRUPT; + lfs_unmount(&lfs) => 0; +''' + +[[case]] # invalid file pointer test +in = "lfs.c" +define.SIZE = [10, 1000, 100000] # faked file size +code = ''' + // create littlefs + lfs_format(&lfs, &cfg) => 0; + // make a file + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "file_here", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + + // change the file pointer to be invalid + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + // make sure id 1 == our file + lfs_dir_get(&lfs, &mdir, + LFS_MKTAG(0x700, 0x3ff, 0), + LFS_MKTAG(LFS_TYPE_NAME, 1, strlen("file_here")), buffer) + => LFS_MKTAG(LFS_TYPE_REG, 1, strlen("file_here")); + assert(memcmp((char*)buffer, "file_here", strlen("file_here")) == 0); + // change file pointer + lfs_dir_commit(&lfs, &mdir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_CTZSTRUCT, 1, sizeof(struct lfs_ctz)), + &(struct lfs_ctz){0xcccccccc, lfs_tole32(SIZE)}})) => 0; + lfs_deinit(&lfs) => 0; + + // test that accessing our bad file fails, note there's a number + // of ways to access the dir, some can fail, but some don't + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "file_here", &info) => 0; + assert(strcmp(info.name, "file_here") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + + lfs_file_open(&lfs, &file, "file_here", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, SIZE) => LFS_ERR_CORRUPT; + lfs_file_close(&lfs, &file) => 0; + + // any allocs that traverse CTZ must unfortunately must fail + if (SIZE > 2*LFS_BLOCK_SIZE) { + lfs_mkdir(&lfs, "dir_here") => LFS_ERR_CORRUPT; + } + lfs_unmount(&lfs) => 0; +''' + +[[case]] # invalid pointer in CTZ skip-list test +define.SIZE = ['2*LFS_BLOCK_SIZE', '3*LFS_BLOCK_SIZE', '4*LFS_BLOCK_SIZE'] +in = "lfs.c" +code = ''' + // create littlefs + lfs_format(&lfs, &cfg) => 0; + // make a file + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "file_here", + LFS_O_WRONLY | LFS_O_CREAT) => 0; + for (int i = 0; i < SIZE; i++) { + char c = 'c'; + lfs_file_write(&lfs, &file, &c, 1) => 1; + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; + // change pointer in CTZ skip-list to be invalid + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + // make sure id 1 == our file and get our CTZ structure + lfs_dir_get(&lfs, &mdir, + LFS_MKTAG(0x700, 0x3ff, 0), + LFS_MKTAG(LFS_TYPE_NAME, 1, strlen("file_here")), buffer) + => LFS_MKTAG(LFS_TYPE_REG, 1, strlen("file_here")); + assert(memcmp((char*)buffer, "file_here", strlen("file_here")) == 0); + struct lfs_ctz ctz; + lfs_dir_get(&lfs, &mdir, + LFS_MKTAG(0x700, 0x3ff, 0), + LFS_MKTAG(LFS_TYPE_STRUCT, 1, sizeof(struct lfs_ctz)), &ctz) + => LFS_MKTAG(LFS_TYPE_CTZSTRUCT, 1, sizeof(struct lfs_ctz)); + lfs_ctz_fromle32(&ctz); + // rewrite block to contain bad pointer + uint8_t bbuffer[LFS_BLOCK_SIZE]; + cfg.read(&cfg, ctz.head, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + uint32_t bad = lfs_tole32(0xcccccccc); + memcpy(&bbuffer[0], &bad, sizeof(bad)); + memcpy(&bbuffer[4], &bad, sizeof(bad)); + cfg.erase(&cfg, ctz.head) => 0; + cfg.prog(&cfg, ctz.head, 0, bbuffer, LFS_BLOCK_SIZE) => 0; + lfs_deinit(&lfs) => 0; + + // test that accessing our bad file fails, note there's a number + // of ways to access the dir, some can fail, but some don't + lfs_mount(&lfs, &cfg) => 0; + lfs_stat(&lfs, "file_here", &info) => 0; + assert(strcmp(info.name, "file_here") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + + lfs_file_open(&lfs, &file, "file_here", LFS_O_RDONLY) => 0; + lfs_file_read(&lfs, &file, buffer, SIZE) => LFS_ERR_CORRUPT; + lfs_file_close(&lfs, &file) => 0; + + // any allocs that traverse CTZ must unfortunately must fail + if (SIZE > 2*LFS_BLOCK_SIZE) { + lfs_mkdir(&lfs, "dir_here") => LFS_ERR_CORRUPT; + } + lfs_unmount(&lfs) => 0; +''' + + +[[case]] # invalid gstate pointer +define.INVALSET = [0x3, 0x1, 0x2] +in = "lfs.c" +code = ''' + // create littlefs + lfs_format(&lfs, &cfg) => 0; + + // create an invalid gstate + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + lfs_fs_prepmove(&lfs, 1, (lfs_block_t [2]){ + (INVALSET & 0x1) ? 0xcccccccc : 0, + (INVALSET & 0x2) ? 0xcccccccc : 0}); + lfs_dir_commit(&lfs, &mdir, NULL, 0) => 0; + lfs_deinit(&lfs) => 0; + + // test that mount fails gracefully + // mount may not fail, but our first alloc should fail when + // we try to fix the gstate + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "should_fail") => LFS_ERR_CORRUPT; + lfs_unmount(&lfs) => 0; +''' + +# cycle detection/recovery tests + +[[case]] # metadata-pair threaded-list loop test +in = "lfs.c" +code = ''' + // create littlefs + lfs_format(&lfs, &cfg) => 0; + + // change tail-pointer to point to ourself + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + lfs_dir_commit(&lfs, &mdir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_HARDTAIL, 0x3ff, 8), + (lfs_block_t[2]){0, 1}})) => 0; + lfs_deinit(&lfs) => 0; + + // test that mount fails gracefully + lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; +''' + +[[case]] # metadata-pair threaded-list 2-length loop test +in = "lfs.c" +code = ''' + // create littlefs with child dir + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "child") => 0; + lfs_unmount(&lfs) => 0; + + // find child + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_block_t pair[2]; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + lfs_dir_get(&lfs, &mdir, + LFS_MKTAG(0x7ff, 0x3ff, 0), + LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 1, sizeof(pair)), pair) + => LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 1, sizeof(pair)); + lfs_pair_fromle32(pair); + // change tail-pointer to point to root + lfs_dir_fetch(&lfs, &mdir, pair) => 0; + lfs_dir_commit(&lfs, &mdir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_HARDTAIL, 0x3ff, 8), + (lfs_block_t[2]){0, 1}})) => 0; + lfs_deinit(&lfs) => 0; + + // test that mount fails gracefully + lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; +''' + +[[case]] # metadata-pair threaded-list 1-length child loop test +in = "lfs.c" +code = ''' + // create littlefs with child dir + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + lfs_mkdir(&lfs, "child") => 0; + lfs_unmount(&lfs) => 0; + + // find child + lfs_init(&lfs, &cfg) => 0; + lfs_mdir_t mdir; + lfs_block_t pair[2]; + lfs_dir_fetch(&lfs, &mdir, (lfs_block_t[2]){0, 1}) => 0; + lfs_dir_get(&lfs, &mdir, + LFS_MKTAG(0x7ff, 0x3ff, 0), + LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 1, sizeof(pair)), pair) + => LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 1, sizeof(pair)); + lfs_pair_fromle32(pair); + // change tail-pointer to point to ourself + lfs_dir_fetch(&lfs, &mdir, pair) => 0; + lfs_dir_commit(&lfs, &mdir, LFS_MKATTRS( + {LFS_MKTAG(LFS_TYPE_HARDTAIL, 0x3ff, 8), pair})) => 0; + lfs_deinit(&lfs) => 0; + + // test that mount fails gracefully + lfs_mount(&lfs, &cfg) => LFS_ERR_CORRUPT; +''' From d498b9fb31504af2d2e580d3ec0a5031cdd5d941 Mon Sep 17 00:00:00 2001 From: Derek Thrasher Date: Tue, 24 Mar 2020 09:22:05 -0400 Subject: [PATCH 35/41] (bugfix) adding line function to clear out all the global 'free' information so that we can reset it after a failed traversal --- lfs.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/lfs.c b/lfs.c index 93a124d4..4caa363a 100644 --- a/lfs.c +++ b/lfs.c @@ -10,6 +10,8 @@ #define LFS_BLOCK_NULL ((lfs_block_t)-1) #define LFS_BLOCK_INLINE ((lfs_block_t)-2) +static void lfs_alloc_ack(lfs_t *lfs); + /// Caching block device operations /// static inline void lfs_cache_drop(lfs_t *lfs, lfs_cache_t *rcache) { // do not zero, cheaper if cache is readonly or only going to be @@ -24,6 +26,15 @@ static inline void lfs_cache_zero(lfs_t *lfs, lfs_cache_t *pcache) { pcache->block = LFS_BLOCK_NULL; } +/// Invalidate the lookahead buffer. This is done during mounting and failed traversals /// +static inline void lfs_setup_invalid_lookahead_buffer(lfs_t *lfs) +{ + lfs->free.off = lfs->seed % lfs->cfg->block_size; + lfs->free.size = 0; + lfs->free.i = 0; + lfs_alloc_ack(lfs); +} + static int lfs_bd_read(lfs_t *lfs, const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint, lfs_block_t block, lfs_off_t off, @@ -477,6 +488,7 @@ static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) { memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size); int err = lfs_fs_traverseraw(lfs, lfs_alloc_lookahead, lfs, true); if (err) { + lfs_setup_invalid_lookahead_buffer(lfs); return err; } } @@ -3772,10 +3784,7 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { lfs->gdisk = lfs->gstate; // setup free lookahead - lfs->free.off = lfs->seed % lfs->cfg->block_size; - lfs->free.size = 0; - lfs->free.i = 0; - lfs_alloc_ack(lfs); + lfs_setup_invalid_lookahead_buffer(lfs); LFS_TRACE("lfs_mount -> %d", 0); return 0; @@ -4594,6 +4603,7 @@ static int lfs1_mount(lfs_t *lfs, struct lfs1 *lfs1, lfs->lfs1->root[1] = LFS_BLOCK_NULL; // setup free lookahead + // TODO should this also call lfs_setup_invalid_lookahead_buffer(lfs); the free.off is different in the current version of lfs lfs->free.off = 0; lfs->free.size = 0; lfs->free.i = 0; From 5e5b5d85724607b79efb426c02da3e2009e607b1 Mon Sep 17 00:00:00 2001 From: Derek Thrasher Date: Thu, 26 Mar 2020 08:47:12 -0400 Subject: [PATCH 36/41] (chore) updates from PR, we decided not to move forward with changing v1 code since it can be risky. Let's improve the future! Also renamed and moved around a the lookahead free / reset function --- lfs.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/lfs.c b/lfs.c index 4caa363a..72b82208 100644 --- a/lfs.c +++ b/lfs.c @@ -26,15 +26,6 @@ static inline void lfs_cache_zero(lfs_t *lfs, lfs_cache_t *pcache) { pcache->block = LFS_BLOCK_NULL; } -/// Invalidate the lookahead buffer. This is done during mounting and failed traversals /// -static inline void lfs_setup_invalid_lookahead_buffer(lfs_t *lfs) -{ - lfs->free.off = lfs->seed % lfs->cfg->block_size; - lfs->free.size = 0; - lfs->free.i = 0; - lfs_alloc_ack(lfs); -} - static int lfs_bd_read(lfs_t *lfs, const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint, lfs_block_t block, lfs_off_t off, @@ -448,6 +439,19 @@ static int lfs_alloc_lookahead(void *p, lfs_block_t block) { return 0; } +static void lfs_alloc_ack(lfs_t *lfs) { + lfs->free.ack = lfs->cfg->block_count; +} + +/// Invalidate the lookahead buffer. This is done during mounting and failed traversals /// +static void lfs_alloc_reset(lfs_t *lfs) +{ + lfs->free.off = lfs->seed % lfs->cfg->block_size; + lfs->free.size = 0; + lfs->free.i = 0; + lfs_alloc_ack(lfs); +} + static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) { while (true) { while (lfs->free.i != lfs->free.size) { @@ -488,17 +492,12 @@ static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) { memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size); int err = lfs_fs_traverseraw(lfs, lfs_alloc_lookahead, lfs, true); if (err) { - lfs_setup_invalid_lookahead_buffer(lfs); + lfs_alloc_reset(lfs); return err; } } } -static void lfs_alloc_ack(lfs_t *lfs) { - lfs->free.ack = lfs->cfg->block_count; -} - - /// Metadata pair and directory operations /// static lfs_stag_t lfs_dir_getslice(lfs_t *lfs, const lfs_mdir_t *dir, lfs_tag_t gmask, lfs_tag_t gtag, @@ -3784,7 +3783,7 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { lfs->gdisk = lfs->gstate; // setup free lookahead - lfs_setup_invalid_lookahead_buffer(lfs); + lfs_alloc_reset(lfs); LFS_TRACE("lfs_mount -> %d", 0); return 0; @@ -4603,7 +4602,6 @@ static int lfs1_mount(lfs_t *lfs, struct lfs1 *lfs1, lfs->lfs1->root[1] = LFS_BLOCK_NULL; // setup free lookahead - // TODO should this also call lfs_setup_invalid_lookahead_buffer(lfs); the free.off is different in the current version of lfs lfs->free.off = 0; lfs->free.size = 0; lfs->free.i = 0; From f17d3d7ebae42fc97ac740d7cc1a89d80524585d Mon Sep 17 00:00:00 2001 From: Derek Thrasher Date: Thu, 26 Mar 2020 08:48:52 -0400 Subject: [PATCH 37/41] Minor cleanup - Removed the declaration of lfs_alloc_ack - Consistent brackets --- lfs.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/lfs.c b/lfs.c index 72b82208..d7b4a90b 100644 --- a/lfs.c +++ b/lfs.c @@ -10,8 +10,6 @@ #define LFS_BLOCK_NULL ((lfs_block_t)-1) #define LFS_BLOCK_INLINE ((lfs_block_t)-2) -static void lfs_alloc_ack(lfs_t *lfs); - /// Caching block device operations /// static inline void lfs_cache_drop(lfs_t *lfs, lfs_cache_t *rcache) { // do not zero, cheaper if cache is readonly or only going to be @@ -443,9 +441,9 @@ static void lfs_alloc_ack(lfs_t *lfs) { lfs->free.ack = lfs->cfg->block_count; } -/// Invalidate the lookahead buffer. This is done during mounting and failed traversals /// -static void lfs_alloc_reset(lfs_t *lfs) -{ +// Invalidate the lookahead buffer. This is done during mounting and +// failed traversals +static void lfs_alloc_reset(lfs_t *lfs) { lfs->free.off = lfs->seed % lfs->cfg->block_size; lfs->free.size = 0; lfs->free.i = 0; From f9dbec3d92b9b71c81accbcfbf7e8da8f6ab70ea Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Tue, 24 Mar 2020 17:30:25 -0500 Subject: [PATCH 38/41] Added test case catching issues with errors during a lookahead scan Original issue found by thrasher8390 --- tests/test_alloc.toml | 84 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/tests/test_alloc.toml b/tests/test_alloc.toml index c05f001c..fa92da51 100644 --- a/tests/test_alloc.toml +++ b/tests/test_alloc.toml @@ -323,6 +323,90 @@ code = ''' lfs_unmount(&lfs) => 0; ''' +[[case]] # what if we have a bad block during an allocation scan? +in = "lfs.c" +define.LFS_ERASE_CYCLES = 0xffffffff +define.LFS_BADBLOCK_BEHAVIOR = 'LFS_TESTBD_BADBLOCK_READERROR' +code = ''' + lfs_format(&lfs, &cfg) => 0; + lfs_mount(&lfs, &cfg) => 0; + // first fill to exhaustion to find available space + lfs_file_open(&lfs, &file, "pacman", LFS_O_WRONLY | LFS_O_CREAT) => 0; + strcpy((char*)buffer, "waka"); + size = strlen("waka"); + lfs_size_t filesize = 0; + while (true) { + lfs_ssize_t res = lfs_file_write(&lfs, &file, buffer, size); + assert(res == (lfs_ssize_t)size || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + break; + } + filesize += size; + } + lfs_file_close(&lfs, &file) => 0; + // now fill all but a couple of blocks of the filesystem with data + filesize -= 3*LFS_BLOCK_SIZE; + lfs_file_open(&lfs, &file, "pacman", LFS_O_WRONLY | LFS_O_CREAT) => 0; + strcpy((char*)buffer, "waka"); + size = strlen("waka"); + for (lfs_size_t i = 0; i < filesize/size; i++) { + lfs_file_write(&lfs, &file, buffer, size) => size; + } + lfs_file_close(&lfs, &file) => 0; + // also save head of file so we can error during lookahead scan + lfs_block_t fileblock = file.ctz.head; + lfs_unmount(&lfs) => 0; + + // remount to force an alloc scan + lfs_mount(&lfs, &cfg) => 0; + + // but mark the head of our file as a "bad block", this is force our + // scan to bail early + lfs_testbd_setwear(&cfg, fileblock, 0xffffffff) => 0; + lfs_file_open(&lfs, &file, "ghost", LFS_O_WRONLY | LFS_O_CREAT) => 0; + strcpy((char*)buffer, "chomp"); + size = strlen("chomp"); + while (true) { + lfs_ssize_t res = lfs_file_write(&lfs, &file, buffer, size); + assert(res == (lfs_ssize_t)size || res == LFS_ERR_CORRUPT); + if (res == LFS_ERR_CORRUPT) { + break; + } + } + lfs_file_close(&lfs, &file) => 0; + + // now reverse the "bad block" and try to write the file again until we + // run out of space + lfs_testbd_setwear(&cfg, fileblock, 0) => 0; + lfs_file_open(&lfs, &file, "ghost", LFS_O_WRONLY | LFS_O_CREAT) => 0; + strcpy((char*)buffer, "chomp"); + size = strlen("chomp"); + while (true) { + lfs_ssize_t res = lfs_file_write(&lfs, &file, buffer, size); + assert(res == (lfs_ssize_t)size || res == LFS_ERR_NOSPC); + if (res == LFS_ERR_NOSPC) { + break; + } + } + lfs_file_close(&lfs, &file) => 0; + + lfs_unmount(&lfs) => 0; + + // check that the disk isn't hurt + lfs_mount(&lfs, &cfg) => 0; + lfs_file_open(&lfs, &file, "pacman", LFS_O_RDONLY) => 0; + strcpy((char*)buffer, "waka"); + size = strlen("waka"); + for (lfs_size_t i = 0; i < filesize/size; i++) { + uint8_t rbuffer[4]; + lfs_file_read(&lfs, &file, rbuffer, size) => size; + assert(memcmp(rbuffer, buffer, size) == 0); + } + lfs_file_close(&lfs, &file) => 0; + lfs_unmount(&lfs) => 0; +''' + + # Below, I don't like these tests. They're fragile and depend _heavily_ # on the geometry of the block device. But they are valuable. Eventually they # should be removed and replaced with generalized tests. From ff84902970326efc4449f5cb1a8ccd3b466e18a9 Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 29 Mar 2020 18:45:51 -0500 Subject: [PATCH 39/41] Moved out block device tracing into separate define Block device tracing has a lot of potential uses, of course debugging, but it can also be used for profiling and externally tracking littlefs's usage of the block device. However, block device tracing emits a massive amount of output. So keeping block device tracing on by default limits the usefulness of the filesystem tracing. So, instead, I've moved the block device tracing into a separate LFS_TESTBD_YES_TRACE define which switches on the LFS_TESTBD_TRACE macro. Note that this means in order to get block device tracing, you need to define both LFS_YES_TRACE and LFS_TESTBD_YES_TRACE. This is needed as the LFS_TRACE definition is gated by LFS_YES_TRACE in lfs_util.h. --- bd/lfs_filebd.c | 51 ++++++++++++++++++++--------------------- bd/lfs_filebd.h | 8 +++++++ bd/lfs_rambd.c | 32 +++++++++++++------------- bd/lfs_rambd.h | 8 +++++++ bd/lfs_testbd.c | 60 +++++++++++++++++++++++++------------------------ bd/lfs_testbd.h | 7 ++++++ 6 files changed, 97 insertions(+), 69 deletions(-) diff --git a/bd/lfs_filebd.c b/bd/lfs_filebd.c index 01ba7014..2d36a42f 100644 --- a/bd/lfs_filebd.c +++ b/bd/lfs_filebd.c @@ -12,7 +12,7 @@ int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, const struct lfs_filebd_config *bdcfg) { - LFS_TRACE("lfs_filebd_createcfg(%p {.context=%p, " + LFS_FILEBD_TRACE("lfs_filebd_createcfg(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " @@ -30,16 +30,16 @@ int lfs_filebd_createcfg(const struct lfs_config *cfg, const char *path, bd->fd = open(path, O_RDWR | O_CREAT, 0666); if (bd->fd < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_createcfg -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_createcfg -> %d", err); return err; } - LFS_TRACE("lfs_filebd_createcfg -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_createcfg -> %d", 0); return 0; } int lfs_filebd_create(const struct lfs_config *cfg, const char *path) { - LFS_TRACE("lfs_filebd_create(%p {.context=%p, " + LFS_FILEBD_TRACE("lfs_filebd_create(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " @@ -51,26 +51,27 @@ int lfs_filebd_create(const struct lfs_config *cfg, const char *path) { path); static const struct lfs_filebd_config defaults = {.erase_value=-1}; int err = lfs_filebd_createcfg(cfg, path, &defaults); - LFS_TRACE("lfs_filebd_create -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_create -> %d", err); return err; } int lfs_filebd_destroy(const struct lfs_config *cfg) { - LFS_TRACE("lfs_filebd_destroy(%p)", (void*)cfg); + LFS_FILEBD_TRACE("lfs_filebd_destroy(%p)", (void*)cfg); lfs_filebd_t *bd = cfg->context; int err = close(bd->fd); if (err < 0) { err = -errno; - LFS_TRACE("lfs_filebd_destroy -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_destroy -> %d", err); return err; } - LFS_TRACE("lfs_filebd_destroy -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_destroy -> %d", 0); return 0; } int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_filebd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + LFS_FILEBD_TRACE("lfs_filebd_read(%p, " + "0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", (void*)cfg, block, off, buffer, size); lfs_filebd_t *bd = cfg->context; @@ -89,24 +90,24 @@ int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block, (off_t)block*cfg->block_size + (off_t)off, SEEK_SET); if (res1 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_read -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_read -> %d", err); return err; } ssize_t res2 = read(bd->fd, buffer, size); if (res2 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_read -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_read -> %d", err); return err; } - LFS_TRACE("lfs_filebd_read -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_read -> %d", 0); return 0; } int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block, lfs_off_t off, const void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_filebd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + LFS_FILEBD_TRACE("lfs_filebd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", (void*)cfg, block, off, buffer, size); lfs_filebd_t *bd = cfg->context; @@ -121,7 +122,7 @@ int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block, (off_t)block*cfg->block_size + (off_t)off, SEEK_SET); if (res1 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_prog -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_prog -> %d", err); return err; } @@ -130,7 +131,7 @@ int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block, ssize_t res2 = read(bd->fd, &c, 1); if (res2 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_prog -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_prog -> %d", err); return err; } @@ -143,23 +144,23 @@ int lfs_filebd_prog(const struct lfs_config *cfg, lfs_block_t block, (off_t)block*cfg->block_size + (off_t)off, SEEK_SET); if (res1 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_prog -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_prog -> %d", err); return err; } ssize_t res2 = write(bd->fd, buffer, size); if (res2 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_prog -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_prog -> %d", err); return err; } - LFS_TRACE("lfs_filebd_prog -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_prog -> %d", 0); return 0; } int lfs_filebd_erase(const struct lfs_config *cfg, lfs_block_t block) { - LFS_TRACE("lfs_filebd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); + LFS_FILEBD_TRACE("lfs_filebd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); lfs_filebd_t *bd = cfg->context; // check if erase is valid @@ -170,7 +171,7 @@ int lfs_filebd_erase(const struct lfs_config *cfg, lfs_block_t block) { off_t res1 = lseek(bd->fd, (off_t)block*cfg->block_size, SEEK_SET); if (res1 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_erase -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_erase -> %d", err); return err; } @@ -178,27 +179,27 @@ int lfs_filebd_erase(const struct lfs_config *cfg, lfs_block_t block) { ssize_t res2 = write(bd->fd, &(uint8_t){bd->cfg->erase_value}, 1); if (res2 < 0) { int err = -errno; - LFS_TRACE("lfs_filebd_erase -> %d", err); + LFS_FILEBD_TRACE("lfs_filebd_erase -> %d", err); return err; } } } - LFS_TRACE("lfs_filebd_erase -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_erase -> %d", 0); return 0; } int lfs_filebd_sync(const struct lfs_config *cfg) { - LFS_TRACE("lfs_filebd_sync(%p)", (void*)cfg); + LFS_FILEBD_TRACE("lfs_filebd_sync(%p)", (void*)cfg); // file sync lfs_filebd_t *bd = cfg->context; int err = fsync(bd->fd); if (err) { err = -errno; - LFS_TRACE("lfs_filebd_sync -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_sync -> %d", 0); return err; } - LFS_TRACE("lfs_filebd_sync -> %d", 0); + LFS_FILEBD_TRACE("lfs_filebd_sync -> %d", 0); return 0; } diff --git a/bd/lfs_filebd.h b/bd/lfs_filebd.h index 271e873d..0d56434a 100644 --- a/bd/lfs_filebd.h +++ b/bd/lfs_filebd.h @@ -15,6 +15,14 @@ extern "C" { #endif + +// Block device specific tracing +#ifdef LFS_FILEBD_YES_TRACE +#define LFS_FILEBD_TRACE(...) LFS_TRACE(__VA_ARGS__) +#else +#define LFS_FILEBD_TRACE(...) +#endif + // filebd config (optional) struct lfs_filebd_config { // 8-bit erase value to use for simulating erases. -1 does not simulate diff --git a/bd/lfs_rambd.c b/bd/lfs_rambd.c index c7c507c4..0a6b5cca 100644 --- a/bd/lfs_rambd.c +++ b/bd/lfs_rambd.c @@ -8,7 +8,7 @@ int lfs_rambd_createcfg(const struct lfs_config *cfg, const struct lfs_rambd_config *bdcfg) { - LFS_TRACE("lfs_rambd_createcfg(%p {.context=%p, " + LFS_RAMBD_TRACE("lfs_rambd_createcfg(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " @@ -27,7 +27,7 @@ int lfs_rambd_createcfg(const struct lfs_config *cfg, } else { bd->buffer = lfs_malloc(cfg->block_size * cfg->block_count); if (!bd->buffer) { - LFS_TRACE("lfs_rambd_createcfg -> %d", LFS_ERR_NOMEM); + LFS_RAMBD_TRACE("lfs_rambd_createcfg -> %d", LFS_ERR_NOMEM); return LFS_ERR_NOMEM; } } @@ -38,12 +38,12 @@ int lfs_rambd_createcfg(const struct lfs_config *cfg, cfg->block_size * cfg->block_count); } - LFS_TRACE("lfs_rambd_createcfg -> %d", 0); + LFS_RAMBD_TRACE("lfs_rambd_createcfg -> %d", 0); return 0; } int lfs_rambd_create(const struct lfs_config *cfg) { - LFS_TRACE("lfs_rambd_create(%p {.context=%p, " + LFS_RAMBD_TRACE("lfs_rambd_create(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " ".block_size=%"PRIu32", .block_count=%"PRIu32"})", @@ -53,24 +53,25 @@ int lfs_rambd_create(const struct lfs_config *cfg) { cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count); static const struct lfs_rambd_config defaults = {.erase_value=-1}; int err = lfs_rambd_createcfg(cfg, &defaults); - LFS_TRACE("lfs_rambd_create -> %d", err); + LFS_RAMBD_TRACE("lfs_rambd_create -> %d", err); return err; } int lfs_rambd_destroy(const struct lfs_config *cfg) { - LFS_TRACE("lfs_rambd_destroy(%p)", (void*)cfg); + LFS_RAMBD_TRACE("lfs_rambd_destroy(%p)", (void*)cfg); // clean up memory lfs_rambd_t *bd = cfg->context; if (!bd->cfg->buffer) { lfs_free(bd->buffer); } - LFS_TRACE("lfs_rambd_destroy -> %d", 0); + LFS_RAMBD_TRACE("lfs_rambd_destroy -> %d", 0); return 0; } int lfs_rambd_read(const struct lfs_config *cfg, lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_rambd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + LFS_RAMBD_TRACE("lfs_rambd_read(%p, " + "0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", (void*)cfg, block, off, buffer, size); lfs_rambd_t *bd = cfg->context; @@ -82,13 +83,14 @@ int lfs_rambd_read(const struct lfs_config *cfg, lfs_block_t block, // read data memcpy(buffer, &bd->buffer[block*cfg->block_size + off], size); - LFS_TRACE("lfs_rambd_read -> %d", 0); + LFS_RAMBD_TRACE("lfs_rambd_read -> %d", 0); return 0; } int lfs_rambd_prog(const struct lfs_config *cfg, lfs_block_t block, lfs_off_t off, const void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_rambd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + LFS_RAMBD_TRACE("lfs_rambd_prog(%p, " + "0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", (void*)cfg, block, off, buffer, size); lfs_rambd_t *bd = cfg->context; @@ -108,12 +110,12 @@ int lfs_rambd_prog(const struct lfs_config *cfg, lfs_block_t block, // program data memcpy(&bd->buffer[block*cfg->block_size + off], buffer, size); - LFS_TRACE("lfs_rambd_prog -> %d", 0); + LFS_RAMBD_TRACE("lfs_rambd_prog -> %d", 0); return 0; } int lfs_rambd_erase(const struct lfs_config *cfg, lfs_block_t block) { - LFS_TRACE("lfs_rambd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); + LFS_RAMBD_TRACE("lfs_rambd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); lfs_rambd_t *bd = cfg->context; // check if erase is valid @@ -125,14 +127,14 @@ int lfs_rambd_erase(const struct lfs_config *cfg, lfs_block_t block) { bd->cfg->erase_value, cfg->block_size); } - LFS_TRACE("lfs_rambd_erase -> %d", 0); + LFS_RAMBD_TRACE("lfs_rambd_erase -> %d", 0); return 0; } int lfs_rambd_sync(const struct lfs_config *cfg) { - LFS_TRACE("lfs_rambd_sync(%p)", (void*)cfg); + LFS_RAMBD_TRACE("lfs_rambd_sync(%p)", (void*)cfg); // sync does nothing because we aren't backed by anything real (void)cfg; - LFS_TRACE("lfs_rambd_sync -> %d", 0); + LFS_RAMBD_TRACE("lfs_rambd_sync -> %d", 0); return 0; } diff --git a/bd/lfs_rambd.h b/bd/lfs_rambd.h index 139ddd67..56a45ce9 100644 --- a/bd/lfs_rambd.h +++ b/bd/lfs_rambd.h @@ -15,6 +15,14 @@ extern "C" { #endif + +// Block device specific tracing +#ifdef LFS_RAMBD_YES_TRACE +#define LFS_RAMBD_TRACE(...) LFS_TRACE(__VA_ARGS__) +#else +#define LFS_RAMBD_TRACE(...) +#endif + // rambd config (optional) struct lfs_rambd_config { // 8-bit erase value to simulate erasing with. -1 indicates no erase diff --git a/bd/lfs_testbd.c b/bd/lfs_testbd.c index 8de7dcbc..1ec6fb90 100644 --- a/bd/lfs_testbd.c +++ b/bd/lfs_testbd.c @@ -12,7 +12,7 @@ int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, const struct lfs_testbd_config *bdcfg) { - LFS_TRACE("lfs_testbd_createcfg(%p {.context=%p, " + LFS_TESTBD_TRACE("lfs_testbd_createcfg(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " @@ -38,9 +38,9 @@ int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, if (bd->cfg->wear_buffer) { bd->wear = bd->cfg->wear_buffer; } else { - bd->wear = lfs_malloc(sizeof(lfs_testbd_wear_t) * cfg->block_count); + bd->wear = lfs_malloc(sizeof(lfs_testbd_wear_t)*cfg->block_count); if (!bd->wear) { - LFS_TRACE("lfs_testbd_createcfg -> %d", LFS_ERR_NOMEM); + LFS_TESTBD_TRACE("lfs_testbd_createcfg -> %d", LFS_ERR_NOMEM); return LFS_ERR_NOMEM; } } @@ -54,7 +54,7 @@ int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, .erase_value = bd->cfg->erase_value, }; int err = lfs_filebd_createcfg(cfg, path, &bd->u.file.cfg); - LFS_TRACE("lfs_testbd_createcfg -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_createcfg -> %d", err); return err; } else { bd->u.ram.cfg = (struct lfs_rambd_config){ @@ -62,13 +62,13 @@ int lfs_testbd_createcfg(const struct lfs_config *cfg, const char *path, .buffer = bd->cfg->buffer, }; int err = lfs_rambd_createcfg(cfg, &bd->u.ram.cfg); - LFS_TRACE("lfs_testbd_createcfg -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_createcfg -> %d", err); return err; } } int lfs_testbd_create(const struct lfs_config *cfg, const char *path) { - LFS_TRACE("lfs_testbd_create(%p {.context=%p, " + LFS_TESTBD_TRACE("lfs_testbd_create(%p {.context=%p, " ".read=%p, .prog=%p, .erase=%p, .sync=%p, " ".read_size=%"PRIu32", .prog_size=%"PRIu32", " ".block_size=%"PRIu32", .block_count=%"PRIu32"}, " @@ -80,12 +80,12 @@ int lfs_testbd_create(const struct lfs_config *cfg, const char *path) { path); static const struct lfs_testbd_config defaults = {.erase_value=-1}; int err = lfs_testbd_createcfg(cfg, path, &defaults); - LFS_TRACE("lfs_testbd_create -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_create -> %d", err); return err; } int lfs_testbd_destroy(const struct lfs_config *cfg) { - LFS_TRACE("lfs_testbd_destroy(%p)", (void*)cfg); + LFS_TESTBD_TRACE("lfs_testbd_destroy(%p)", (void*)cfg); lfs_testbd_t *bd = cfg->context; if (bd->cfg->erase_cycles && !bd->cfg->wear_buffer) { lfs_free(bd->wear); @@ -93,11 +93,11 @@ int lfs_testbd_destroy(const struct lfs_config *cfg) { if (bd->persist) { int err = lfs_filebd_destroy(cfg); - LFS_TRACE("lfs_testbd_destroy -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_destroy -> %d", err); return err; } else { int err = lfs_rambd_destroy(cfg); - LFS_TRACE("lfs_testbd_destroy -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_destroy -> %d", err); return err; } } @@ -145,7 +145,8 @@ static int lfs_testbd_rawsync(const struct lfs_config *cfg) { /// block device API /// int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_testbd_read(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + LFS_TESTBD_TRACE("lfs_testbd_read(%p, " + "0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", (void*)cfg, block, off, buffer, size); lfs_testbd_t *bd = cfg->context; @@ -157,19 +158,20 @@ int lfs_testbd_read(const struct lfs_config *cfg, lfs_block_t block, // block bad? if (bd->cfg->erase_cycles && bd->wear[block] >= bd->cfg->erase_cycles && bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_READERROR) { - LFS_TRACE("lfs_testbd_read -> %d", LFS_ERR_CORRUPT); + LFS_TESTBD_TRACE("lfs_testbd_read -> %d", LFS_ERR_CORRUPT); return LFS_ERR_CORRUPT; } // read int err = lfs_testbd_rawread(cfg, block, off, buffer, size); - LFS_TRACE("lfs_testbd_read -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_read -> %d", err); return err; } int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, lfs_off_t off, const void *buffer, lfs_size_t size) { - LFS_TRACE("lfs_testbd_prog(%p, 0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", + LFS_TESTBD_TRACE("lfs_testbd_prog(%p, " + "0x%"PRIx32", %"PRIu32", %p, %"PRIu32")", (void*)cfg, block, off, buffer, size); lfs_testbd_t *bd = cfg->context; @@ -182,13 +184,13 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, if (bd->cfg->erase_cycles && bd->wear[block] >= bd->cfg->erase_cycles) { if (bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_PROGERROR) { - LFS_TRACE("lfs_testbd_prog -> %d", LFS_ERR_CORRUPT); + LFS_TESTBD_TRACE("lfs_testbd_prog -> %d", LFS_ERR_CORRUPT); return LFS_ERR_CORRUPT; } else if (bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_PROGNOOP || bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_ERASENOOP) { - LFS_TRACE("lfs_testbd_prog -> %d", 0); + LFS_TESTBD_TRACE("lfs_testbd_prog -> %d", 0); return 0; } } @@ -196,7 +198,7 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, // prog int err = lfs_testbd_rawprog(cfg, block, off, buffer, size); if (err) { - LFS_TRACE("lfs_testbd_prog -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_prog -> %d", err); return err; } @@ -211,12 +213,12 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block, } } - LFS_TRACE("lfs_testbd_prog -> %d", 0); + LFS_TESTBD_TRACE("lfs_testbd_prog -> %d", 0); return 0; } int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { - LFS_TRACE("lfs_testbd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); + LFS_TESTBD_TRACE("lfs_testbd_erase(%p, 0x%"PRIx32")", (void*)cfg, block); lfs_testbd_t *bd = cfg->context; // check if erase is valid @@ -227,11 +229,11 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { if (bd->wear[block] >= bd->cfg->erase_cycles) { if (bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_ERASEERROR) { - LFS_TRACE("lfs_testbd_erase -> %d", LFS_ERR_CORRUPT); + LFS_TESTBD_TRACE("lfs_testbd_erase -> %d", LFS_ERR_CORRUPT); return LFS_ERR_CORRUPT; } else if (bd->cfg->badblock_behavior == LFS_TESTBD_BADBLOCK_ERASENOOP) { - LFS_TRACE("lfs_testbd_erase -> %d", 0); + LFS_TESTBD_TRACE("lfs_testbd_erase -> %d", 0); return 0; } } else { @@ -243,7 +245,7 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { // erase int err = lfs_testbd_rawerase(cfg, block); if (err) { - LFS_TRACE("lfs_testbd_erase -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_erase -> %d", err); return err; } @@ -258,14 +260,14 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) { } } - LFS_TRACE("lfs_testbd_prog -> %d", 0); + LFS_TESTBD_TRACE("lfs_testbd_prog -> %d", 0); return 0; } int lfs_testbd_sync(const struct lfs_config *cfg) { - LFS_TRACE("lfs_testbd_sync(%p)", (void*)cfg); + LFS_TESTBD_TRACE("lfs_testbd_sync(%p)", (void*)cfg); int err = lfs_testbd_rawsync(cfg); - LFS_TRACE("lfs_testbd_sync -> %d", err); + LFS_TESTBD_TRACE("lfs_testbd_sync -> %d", err); return err; } @@ -273,20 +275,20 @@ int lfs_testbd_sync(const struct lfs_config *cfg) { /// simulated wear operations /// lfs_testbd_swear_t lfs_testbd_getwear(const struct lfs_config *cfg, lfs_block_t block) { - LFS_TRACE("lfs_testbd_getwear(%p, %"PRIu32")", (void*)cfg, block); + LFS_TESTBD_TRACE("lfs_testbd_getwear(%p, %"PRIu32")", (void*)cfg, block); lfs_testbd_t *bd = cfg->context; // check if block is valid LFS_ASSERT(bd->cfg->erase_cycles); LFS_ASSERT(block < cfg->block_count); - LFS_TRACE("lfs_testbd_getwear -> %"PRIu32, bd->wear[block]); + LFS_TESTBD_TRACE("lfs_testbd_getwear -> %"PRIu32, bd->wear[block]); return bd->wear[block]; } int lfs_testbd_setwear(const struct lfs_config *cfg, lfs_block_t block, lfs_testbd_wear_t wear) { - LFS_TRACE("lfs_testbd_setwear(%p, %"PRIu32")", (void*)cfg, block); + LFS_TESTBD_TRACE("lfs_testbd_setwear(%p, %"PRIu32")", (void*)cfg, block); lfs_testbd_t *bd = cfg->context; // check if block is valid @@ -295,6 +297,6 @@ int lfs_testbd_setwear(const struct lfs_config *cfg, bd->wear[block] = wear; - LFS_TRACE("lfs_testbd_setwear -> %d", 0); + LFS_TESTBD_TRACE("lfs_testbd_setwear -> %d", 0); return 0; } diff --git a/bd/lfs_testbd.h b/bd/lfs_testbd.h index 68180acc..b1fb2e92 100644 --- a/bd/lfs_testbd.h +++ b/bd/lfs_testbd.h @@ -19,6 +19,13 @@ extern "C" #endif +// Block device specific tracing +#ifdef LFS_TESTBD_YES_TRACE +#define LFS_TESTBD_TRACE(...) LFS_TRACE(__VA_ARGS__) +#else +#define LFS_TESTBD_TRACE(...) +#endif + // Mode determining how "bad blocks" behave during testing. This simulates // some real-world circumstances such as progs not sticking (prog-noop), // a readonly disk (erase-noop), and ECC failures (read-error). From 5137e4b0bab12a2d88c1e960acc1acf0f278563f Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 29 Mar 2020 21:19:33 -0500 Subject: [PATCH 40/41] Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore. --- .gitignore | 2 + lfs.c | 44 ++++++------ scripts/readmdir.py | 18 +++-- scripts/readtree.py | 162 ++++++++++++-------------------------------- scripts/test.py | 4 +- 5 files changed, 83 insertions(+), 147 deletions(-) diff --git a/.gitignore b/.gitignore index 9cafad9c..a6ebc4c3 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,5 @@ blocks/ lfs test.c tests/*.toml.* +scripts/__pycache__ +.gdb_history diff --git a/lfs.c b/lfs.c index d7b4a90b..73b88b14 100644 --- a/lfs.c +++ b/lfs.c @@ -979,7 +979,7 @@ static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs, dir->rev = revs[(r+1)%2]; } - LFS_ERROR("Corrupted dir pair at %"PRIx32" %"PRIx32, + LFS_ERROR("Corrupted dir pair at {0x%"PRIx32", 0x%"PRIx32"}", dir->pair[0], dir->pair[1]); return LFS_ERR_CORRUPT; } @@ -1667,12 +1667,13 @@ static int lfs_dir_compact(lfs_t *lfs, relocated = true; lfs_cache_drop(lfs, &lfs->pcache); if (!tired) { - LFS_DEBUG("Bad block at %"PRIx32, dir->pair[1]); + LFS_DEBUG("Bad block at 0x%"PRIx32, dir->pair[1]); } // can't relocate superblock, filesystem is now frozen if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) { - LFS_WARN("Superblock %"PRIx32" has become unwritable", dir->pair[1]); + LFS_WARN("Superblock 0x%"PRIx32" has become unwritable", + dir->pair[1]); return LFS_ERR_NOSPC; } @@ -1688,7 +1689,8 @@ static int lfs_dir_compact(lfs_t *lfs, if (relocated) { // update references if we relocated - LFS_DEBUG("Relocating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + LFS_DEBUG("Relocating {0x%"PRIx32", 0x%"PRIx32"} " + "-> {0x%"PRIx32", 0x%"PRIx32"}", oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]); int err = lfs_fs_relocate(lfs, oldpair, dir->pair); if (err) { @@ -2311,7 +2313,7 @@ static int lfs_ctz_extend(lfs_t *lfs, } relocate: - LFS_DEBUG("Bad block at %"PRIx32, nblock); + LFS_DEBUG("Bad block at 0x%"PRIx32, nblock); // just clear cache and try a new block lfs_cache_drop(lfs, pcache); @@ -2615,7 +2617,7 @@ static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) { return 0; relocate: - LFS_DEBUG("Bad block at %"PRIx32, nblock); + LFS_DEBUG("Bad block at 0x%"PRIx32, nblock); // just clear cache and try a new block lfs_cache_drop(lfs, &lfs->pcache); @@ -2692,7 +2694,7 @@ static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) { break; relocate: - LFS_DEBUG("Bad block at %"PRIx32, file->block); + LFS_DEBUG("Bad block at 0x%"PRIx32, file->block); err = lfs_file_relocate(lfs, file); if (err) { return err; @@ -3716,7 +3718,7 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { uint16_t minor_version = (0xffff & (superblock.version >> 0)); if ((major_version != LFS_DISK_VERSION_MAJOR || minor_version > LFS_DISK_VERSION_MINOR)) { - LFS_ERROR("Invalid version %"PRIu16".%"PRIu16, + LFS_ERROR("Invalid version v%"PRIu16".%"PRIu16, major_version, minor_version); err = LFS_ERR_INVAL; goto cleanup; @@ -3772,7 +3774,7 @@ int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) { // update littlefs with gstate if (!lfs_gstate_iszero(&lfs->gstate)) { - LFS_DEBUG("Found pending gstate %08"PRIx32" %08"PRIx32" %08"PRIx32, + LFS_DEBUG("Found pending gstate 0x%08"PRIx32"%08"PRIx32"%08"PRIx32, lfs->gstate.tag, lfs->gstate.pair[0], lfs->gstate.pair[1]); @@ -3987,8 +3989,6 @@ static int lfs_fs_relocate(lfs_t *lfs, const lfs_block_t oldpair[2], lfs_block_t newpair[2]) { // update internal root if (lfs_pair_cmp(oldpair, lfs->root) == 0) { - LFS_DEBUG("Relocating root %"PRIx32" %"PRIx32, - newpair[0], newpair[1]); lfs->root[0] = newpair[0]; lfs->root[1] = newpair[1]; } @@ -4024,7 +4024,7 @@ static int lfs_fs_relocate(lfs_t *lfs, if (lfs_gstate_hasmovehere(&lfs->gstate, parent.pair)) { moveid = lfs_tag_id(lfs->gstate.tag); LFS_DEBUG("Fixing move while relocating " - "%"PRIx32" %"PRIx32" %"PRIx16"\n", + "{0x%"PRIx32", 0x%"PRIx32"} 0x%"PRIx16"\n", parent.pair[0], parent.pair[1], moveid); lfs_fs_prepmove(lfs, 0x3ff, NULL); if (moveid < lfs_tag_id(tag)) { @@ -4060,7 +4060,7 @@ static int lfs_fs_relocate(lfs_t *lfs, if (lfs_gstate_hasmovehere(&lfs->gstate, parent.pair)) { moveid = lfs_tag_id(lfs->gstate.tag); LFS_DEBUG("Fixing move while relocating " - "%"PRIx32" %"PRIx32" %"PRIx16"\n", + "{0x%"PRIx32", 0x%"PRIx32"} 0x%"PRIx16"\n", parent.pair[0], parent.pair[1], moveid); lfs_fs_prepmove(lfs, 0x3ff, NULL); } @@ -4101,7 +4101,7 @@ static int lfs_fs_demove(lfs_t *lfs) { } // Fix bad moves - LFS_DEBUG("Fixing move %"PRIx32" %"PRIx32" %"PRIx16, + LFS_DEBUG("Fixing move {0x%"PRIx32", 0x%"PRIx32"} 0x%"PRIx16, lfs->gdisk.pair[0], lfs->gdisk.pair[1], lfs_tag_id(lfs->gdisk.tag)); @@ -4152,7 +4152,7 @@ static int lfs_fs_deorphan(lfs_t *lfs) { if (tag == LFS_ERR_NOENT) { // we are an orphan - LFS_DEBUG("Fixing orphan %"PRIx32" %"PRIx32, + LFS_DEBUG("Fixing orphan {0x%"PRIx32", 0x%"PRIx32"}", pdir.tail[0], pdir.tail[1]); err = lfs_dir_drop(lfs, &pdir, &dir); @@ -4174,8 +4174,8 @@ static int lfs_fs_deorphan(lfs_t *lfs) { if (!lfs_pair_sync(pair, pdir.tail)) { // we have desynced - LFS_DEBUG("Fixing half-orphan " - "%"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + LFS_DEBUG("Fixing half-orphan {0x%"PRIx32", 0x%"PRIx32"} " + "-> {0x%"PRIx32", 0x%"PRIx32"}", pdir.tail[0], pdir.tail[1], pair[0], pair[1]); lfs_pair_tole32(pair); @@ -4438,7 +4438,7 @@ static int lfs1_dir_fetch(lfs_t *lfs, } if (!valid) { - LFS_ERROR("Corrupted dir pair at %" PRIx32 " %" PRIx32 , + LFS_ERROR("Corrupted dir pair at {0x%"PRIx32", 0x%"PRIx32"}", tpair[0], tpair[1]); return LFS_ERR_CORRUPT; } @@ -4626,7 +4626,8 @@ static int lfs1_mount(lfs_t *lfs, struct lfs1 *lfs1, } if (err || memcmp(superblock.d.magic, "littlefs", 8) != 0) { - LFS_ERROR("Invalid superblock at %d %d", 0, 1); + LFS_ERROR("Invalid superblock at {0x%"PRIx32", 0x%"PRIx32"}", + 0, 1); err = LFS_ERR_CORRUPT; goto cleanup; } @@ -4635,7 +4636,7 @@ static int lfs1_mount(lfs_t *lfs, struct lfs1 *lfs1, uint16_t minor_version = (0xffff & (superblock.d.version >> 0)); if ((major_version != LFS1_DISK_VERSION_MAJOR || minor_version > LFS1_DISK_VERSION_MINOR)) { - LFS_ERROR("Invalid version %d.%d", major_version, minor_version); + LFS_ERROR("Invalid version v%d.%d", major_version, minor_version); err = LFS_ERR_INVAL; goto cleanup; } @@ -4801,7 +4802,8 @@ int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) { // Copy over first block to thread into fs. Unfortunately // if this fails there is not much we can do. - LFS_DEBUG("Migrating %"PRIx32" %"PRIx32" -> %"PRIx32" %"PRIx32, + LFS_DEBUG("Migrating {0x%"PRIx32", 0x%"PRIx32"} " + "-> {0x%"PRIx32", 0x%"PRIx32"}", lfs->root[0], lfs->root[1], dir1.head[0], dir1.head[1]); err = lfs_bd_erase(lfs, dir1.head[1]); diff --git a/scripts/readmdir.py b/scripts/readmdir.py index 75f8b9ac..b6c3dcca 100755 --- a/scripts/readmdir.py +++ b/scripts/readmdir.py @@ -233,8 +233,8 @@ def __bool__(self): def __lt__(self, other): # corrupt blocks don't count - if not self and other: - return True + if not self or not other: + return bool(other) # use sequence arithmetic to avoid overflow return not ((other.rev - self.rev) & 0x80000000) @@ -318,14 +318,24 @@ def main(args): # find most recent pair mdir = MetadataPair(blocks) - print("mdir {%s} rev %d%s%s" % ( + + try: + mdir.tail = mdir[Tag('tail', 0, 0)] + if mdir.tail.size != 8 or mdir.tail.data == 8*b'\xff': + mdir.tail = None + except KeyError: + mdir.tail = None + + print("mdir {%s} rev %d%s%s%s" % ( ', '.join('%#x' % b for b in [args.block1, args.block2] if b is not None), mdir.rev, ' (was %s)' % ', '.join('%d' % m.rev for m in mdir.pair[1:]) if len(mdir.pair) > 1 else '', - ' (corrupted)' if not mdir else '')) + ' (corrupted!)' if not mdir else '', + ' -> {%#x, %#x}' % struct.unpack('= 0: - f2.seek(block * args.block_size) - dat = f2.read(args.block_size) - data.append(dat[4*(ctz(i)+1) if i != 0 else 0:]) - block, = struct.unpack('= ' ' and c <= '~' else '.' - for c in map(chr, data[:8]))) - if not args.no_truncate and len(desc) < 45 - and data is not None else "")) - - if name.is_('superblock') and struct_.is_('inlinestruct'): - f.write( - " block_size %d\n" - " block_count %d\n" - " name_max %d\n" - " file_max %d\n" - " attr_max %d\n" % struct.unpack( - '= ' ' and c <= '~' else '.' - for c in map(chr, tag.data[:8]))) - if not args.no_truncate and len(desc) < 43 else "")) - - if args.no_truncate: - for i in range(0, len(tag.data), 16): - f.write(" %08x: %-47s %-16s\n" % ( - i, ' '.join('%02x' % c for c in tag.data[i:i+16]), - ''.join(c if c >= ' ' and c <= '~' else '.' - for c in map(chr, tag.data[i:i+16])))) - - if args.no_truncate and data is not None: - for i in range(0, len(data), 16): - f.write(" %08x: %-47s %-16s\n" % ( - i, ' '.join('%02x' % c for c in data[i:i+16]), - ''.join(c if c >= ' ' and c <= '~' else '.' - for c in map(chr, data[i:i+16])))) - def main(args): + superblock = None + gstate = b'\0\0\0\0\0\0\0\0\0\0\0\0' + dirs = [] + mdirs = [] + corrupted = [] + cycle = False with open(args.disk, 'rb') as f: - dirs = [] - superblock = None - gstate = b'' - mdirs = [] - cycle = False tail = (args.block1, args.block2) hard = False while True: @@ -144,6 +61,10 @@ def main(args): except KeyError: pass + # corrupted? + if not mdir: + corrupted.append(mdir) + # add to directories mdirs.append(mdir) if mdir.tail is None or not mdir.tail.is_('hardtail'): @@ -178,7 +99,7 @@ def main(args): dir[0].path = path.replace('//', '/') - # dump tree + # print littlefs + version info version = ('?', '?') if superblock: version = tuple(reversed( @@ -187,53 +108,56 @@ def main(args): "data (truncated, if it fits)" if not any([args.no_truncate, args.tags, args.log, args.all]) else "")) - if gstate: - print("gstate 0x%s" % ''.join('%02x' % c for c in gstate)) - tag = Tag(struct.unpack('=%d" % max(tag.size, 1)) - if tag.type: - print(" move dir {%#x, %#x} id %d" % ( - blocks[0], blocks[1], tag.id)) - + # print gstate + print("gstate 0x%s" % ''.join('%02x' % c for c in gstate)) + tag = Tag(struct.unpack('=%d" % max(tag.size, 1)) + if tag.type: + print(" move dir {%#x, %#x} id %d" % ( + blocks[0], blocks[1], tag.id)) + + # print mdir info for i, dir in enumerate(dirs): print("dir %s" % (json.dumps(dir[0].path) if hasattr(dir[0], 'path') else '(orphan)')) for j, mdir in enumerate(dir): - print("mdir {%#x, %#x} rev %d%s" % ( - mdir.blocks[0], mdir.blocks[1], mdir.rev, - ' (corrupted)' if not mdir else '')) + print("mdir {%#x, %#x} rev %d (was %d)%s%s" % ( + mdir.blocks[0], mdir.blocks[1], mdir.rev, mdir.pair[1].rev, + ' (corrupted!)' if not mdir else '', + ' -> {%#x, %#x}' % struct.unpack(' {%#x, %#x} ***" % (cycle[0], cycle[1])) + errcode = 0 + for mdir in corrupted: + errcode = errcode or 1 + print("*** corrupted mdir {%#x, %#x}! ***" % ( + mdir.blocks[0], mdir.blocks[1])) if cycle: - return 2 - elif not all(mdir for dir in dirs for mdir in dir): - return 1 - else: - return 0; + errcode = errcode or 2 + print("*** cycle detected {%#x, %#x}! ***" % ( + cycle[0], cycle[1])) + + return errcode if __name__ == "__main__": import argparse @@ -246,12 +170,10 @@ def main(args): help="Size of a block in bytes.") parser.add_argument('block1', nargs='?', default=0, type=lambda x: int(x, 0), - help="Optional first block address for finding the root.") + help="Optional first block address for finding the superblock.") parser.add_argument('block2', nargs='?', default=1, type=lambda x: int(x, 0), - help="Optional second block address for finding the root.") - parser.add_argument('-t', '--tags', action='store_true', - help="Show metadata tags instead of reconstructing entries.") + help="Optional second block address for finding the superblock.") parser.add_argument('-l', '--log', action='store_true', help="Show tags in log.") parser.add_argument('-a', '--all', action='store_true', diff --git a/scripts/test.py b/scripts/test.py index 61870289..e5869c20 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -231,7 +231,7 @@ def test(self, exec=[], persist=False, cycles=None, ncmd.extend(['-ex', 'r']) if failure.assert_: ncmd.extend(['-ex', 'up 2']) - elif gdb == 'start': + elif gdb == 'main': ncmd.extend([ '-ex', 'b %s:%d' % (self.suite.path, self.code_lineno), '-ex', 'r']) @@ -760,7 +760,7 @@ def main(**args): help="Store disk image in a file.") parser.add_argument('-b', '--build', action='store_true', help="Only build the tests, do not execute.") - parser.add_argument('-g', '--gdb', choices=['init', 'start', 'assert'], + parser.add_argument('-g', '--gdb', choices=['init', 'main', 'assert'], nargs='?', const='assert', help="Drop into gdb on test failure.") parser.add_argument('--no-internal', action='store_true', From 6622f3deee004ff0002a53437ea2b221e18452bf Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Sun, 29 Mar 2020 21:43:58 -0500 Subject: [PATCH 41/41] Bumped minor version to v2.2 --- lfs.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lfs.h b/lfs.h index a7bd1863..35bbbabf 100644 --- a/lfs.h +++ b/lfs.h @@ -21,7 +21,7 @@ extern "C" // Software library version // Major (top-nibble), incremented on backwards incompatible changes // Minor (bottom-nibble), incremented on feature additions -#define LFS_VERSION 0x00020001 +#define LFS_VERSION 0x00020002 #define LFS_VERSION_MAJOR (0xffff & (LFS_VERSION >> 16)) #define LFS_VERSION_MINOR (0xffff & (LFS_VERSION >> 0))