What is a performant way to load a million+ facts? #1604

jeshan · 2022-09-06T06:24:10Z

jeshan
Sep 6, 2022

I have a lot of static data which I store in a file as simple Prolog terms. I'd then assert them with assertz/1.

Loading it seems never ending while I have been able to load it in a few seconds with Sicstus, Eclipse and SWI (latter with read_file_to_terms/3 which uses read_term/3 internally).

For Scryer, I tried the following minimal example:

read_file_to_terms(Path) :-
    open(Path, read, Stream, []),
    repeat,
    read_term(Stream, _Term, []),
    false.

So, what's the recommended way to load a lot of Prolog terms?

Note that I have liberty on how to write the term file.

I don't know if/how library pio could be made to work more efficiently than the above.

pmoura · 2022-09-07T08:15:58Z

pmoura
Sep 7, 2022

As you have the liberty on how to write the term file, why not write a clause file and skip assert all together by simply consulting the file? Some post-processing not possible at the time the file is written?

P.S. Also note that e.g. SWI-Prolog use a foreign library to accelerate predicates such as read_file_to_terms/3; the Prolog version of the predicate is just a fallback in case the foreign library is not available.

3 replies

jeshan Sep 8, 2022
Author

Can you clarify what you mean by a clause file? If you meant like a normal Prolog source file, it loads but it's not any faster as I understood that consult does a lot more work than the simpler read_term/3.

SWI-Prolog use a foreign library to accelerate predicates

That's good to know and is another reason why we should look to make it fast for Scryer too :)

pmoura Sep 8, 2022

A normal Prolog file. If you're not doing any post-processing on the read terms, then you can consult the terms as clauses for static predicates. Depending on the Prolog system, consulting vs read terms & asserting may indeed be faster or slower; worth experimenting with, something you have already done for comparing the times?

jeshan Sep 8, 2022
Author

If you're not doing any post-processing on the read terms

Indeed, I'm not doing that.

consulting vs read terms & asserting may indeed be faster or slower

I have not tested every combination as I gave up, expecting it to complete within a few seconds. I know that for SWI consult wasn't faster. Scryer crashed similar to the other experiment that I mentioned in a recent comment below.

can consult the terms as clauses for static predicates

Can you explain with an example? e.g how would this be consulted (if not consult/1 directly) ?

couple(alice, bob).

language(prolog, [logic, declarative]).

father_son(bob, carl).

UWN · 2022-09-07T08:29:20Z

UWN
Sep 7, 2022

Could you provide concrete data?

ulrich@p0:/opt/gupu/scryer-prolog$ target/release/scryer-prolog -f
?- open('/dev/null',read,Stream), repeat, read_term(Stream, Term, []).
   Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  ... .

So at least for your test case, it is not surprising that Scryer did not terminate.

4 replies

jeshan Sep 8, 2022
Author

You're right: I had a bug in my less-than-ideal example but that's not the only reason it wasn't terminating. With the fixed code:

% elegance definitely not required here!
read_file_to_terms(Path) :-
    open(Path, read, Stream),
    repeat,
    read_term(Stream, Term, []),
    write(Term),
    (Term=end_of_file -> throw('the end'); true),
    false.

We can see that it is in fact taking a lot of time going through the million facts.

jeshan Sep 8, 2022
Author

Could you provide concrete data?

The data file could just be as simple as the following but with a million such examples

couple(alice, bob).

language(prolog, [logic, declarative]).

father_son(bob, carla).

UWN Sep 8, 2022

(It helps a lot to have exactly one such example, maybe just a little program generating these facts.)

jeshan Sep 8, 2022
Author

I was about to post this and then saw your last message!

I didn't want to post this at first because I thought it would distract from the conversation

write_demo_file :-
    open('out/demo_file.pl', write, Stream),
    ((between(1, 1000000, N),
    write_term(Stream, fact(N), []),
    write(Stream, '.\n'),
    false) -> false; close(Stream)).

jeshan · 2022-09-08T05:45:50Z

jeshan
Sep 8, 2022
Author

On another note: I'm experimenting with writing the term file as a single fact with all the data as a single list in the fact's argument. It crashes with Scryer but works well with SWI:

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
Aborted (core dumped)

0 replies

UWN · 2022-09-08T05:50:04Z

UWN
Sep 8, 2022

What I have tired. A million facts f(N).. with

?- open(fiche_1604,write,S),(between(1,1000000,N),portray_clause(S,f(N)),false;close(S)).
   S = '$dropped_value'.

In Scryer that is ~64s vs ~35s in SWI.

ulrich@TU-Wien:~/scryer$ time /opt/gupu/scryer-prolog/target/release/scryer-prolog -f
?- 

real	0m1,022s
user	0m0,236s
sys	0m0,008s
ulrich@TU-Wien:~/scryer$ time /opt/gupu/scryer-prolog/target/release/scryer-prolog -f fiche_1604 
?- 

real	1m7,738s
user	0m59,047s
sys	0m5,149s
ulrich@TU-Wien:~/scryer$ time swi-prolog.swipl -f none
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.15)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- ^D
% halt

real	0m1,766s
user	0m0,276s
sys	0m0,084s
ulrich@TU-Wien:~/scryer$ 
ulrich@TU-Wien:~/scryer$ time swi-prolog.swipl -f none fiche_1604 
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.15)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- ^D
% halt

real	0m35,918s
user	0m34,682s
sys	0m0,219s

4 replies

jeshan Sep 8, 2022
Author

Check my last message. Were you able to load it back quickly?

UWN Sep 8, 2022

Not clear why you just put between(1, 10, N) into write_demo_file/0 which just works. But essentially that's the same as I have shown above. I replaced it with between(1, 1000000,N). Not much of a difference to my test case above.

jeshan Sep 8, 2022
Author

yes, the 10 was to check with a smaller file first. I then tried with a million and will update the comment

UWN Sep 8, 2022

And just for the record, SICStus is here as fast as SWI.

UWN · 2022-09-08T12:19:58Z

UWN
Sep 8, 2022

To summarize: Scryer is currently about a factor of two slower than SWI for consulting facts.

0 replies

jeshan · 2022-09-17T05:43:25Z

jeshan
Sep 17, 2022
Author

If fast loading isn't available yet, should I close this discussion and open a tracking issue instead?

1 reply

UWN Sep 17, 2022

What exactly is the problem? Being a factor two slower than a systems that is more than ten times as old?

jeshan · 2022-09-17T06:05:17Z

jeshan
Sep 17, 2022
Author

Being a factor two slower

Consulting may be twice as slow but I'm not describing the same thing.

What exactly is the problem?

The question in the title still stands: how to quickly load many facts? (whether via consult, read_term or any other way)

slower than a systems that is more than ten times as old

Well, we'd have to get to it some day, even if it's lower priority.

1 reply

triska Sep 17, 2022

Personally, I would phrase it like this:

If you have all these facts in a file, simply consult it directly, as already suggested by @pmoura. For example, if you know the file before starting your program, simply specify the file on the command line:
```
$ scryer-prolog facts.pl
```
Use this if you have many facts, and also if you have few facts: This is a normal way to consult any Prolog program.
If you do not know the file before starting the program, or you want to make the loading dependent on other factors that only become apparent at run-time (such as input from the user etc.), or you do not want to restart the Prolog system etc., you can use consult/1 to consult the facts (and also rules, and in fact any Prolog program that is stored in a file). You can also use consult/1 as an alternative to (1) at any time.
If you do not want to assert the facts, but rather reason about them as Prolog terms, use read/1 or any of the related predicates that start with read.... In this way, the facts do not become part of the clause database, but are available as Prolog terms for reasoning within your program. Use this approach for example if you want to rewrite these terms etc. without asserting them as facts. This is conceptually different from (1) and (2).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

What is a performant way to load a million+ facts? #1604

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

What is a performant way to load a million+ facts? #1604

Replies: 7 comments · 13 replies

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 8, 2022 Author

jeshan Sep 17, 2022 Author

jeshan Sep 17, 2022 Author

Replies: 7 comments 13 replies

jeshan Sep 8, 2022
Author

jeshan Sep 8, 2022
Author

jeshan Sep 8, 2022
Author

jeshan Sep 8, 2022
Author

jeshan Sep 8, 2022
Author

jeshan
Sep 8, 2022
Author

jeshan Sep 8, 2022
Author

jeshan Sep 8, 2022
Author

jeshan
Sep 17, 2022
Author

jeshan
Sep 17, 2022
Author