[Programming in the Large] module qualification

[jjtolton]

My goal is start steering Scryer towards use in production on larger-ish
projects. Multiple team members, team members of different experience
levels, team members who are present at different points in the
lifecycle of the project. One of the first things that comes to mind is
modules and the module system.

Note: I will often use the phrase "namespace" interchangeably
with "module". Please assume when I'm talking about Prolog and I say
"namespace", I am referring to the general concept of associating
terms with a module and that the technical version of doing that is by
using modules.

Major Points

1. We need a way to alias modules via use_module/3 or something
   similar, i.e.: use_module(Module, Alias, Specifics)
2. We should get in the habit of qualifying everything except the
   builtins, and if we aren't doing that, we should probably start
   treating them as builtins unless a specific restrictive flag is
   present during startup

Unknowns

• Chesteron's Fences regarding modules (is there a reason why we
  aren't doing these practices?)
• ISO/IEC 13211 concepts on modules

Overview

All production languages (with the notable exception of C, in which the
lack of modules/namespaces leads to very verbose names) which are used
for large scale system processing use module or namespace qualification
when writing application code. Even C# and Java, which typically have
using or require directives which alias a class to just the
classname, still count as namespaced because they are namespaced to the
class.

In Python, for instance, it is considered very poor practice do the
following (except in very specific circumstances):

from somelib import *

where everything in the somelib module is now available in the
importing module.

Likewise, in Clojure, it is considered poor practice to do

(ns this-ns
  (:require [somelib :refer :all]))

where likewise all of the somelib namespace is now available.

In these instances, particularly because the languages are dynamic, the
last import wins and overwrites any conflicting names in earlier
imports. Worse, readability suffers.

When there are a significant number of from X import * statements at
the top of a Python module, it can become extremely painful to trace
code -- so much so that you will almost never see from X import * in
ANY (application level) Python code. (The only time it is generally
used is in packaging code to provide a flattened namespace hierarchy).

The preferred options are:

Python

import something.foo as foo

>>> foo.bar(a,b)
>>> foo.some_number + 2

Clojure

(require '[something.foo :as foo])
(foo/bar a b)
(+ foo/some-number 2)

Python and Clojure have different resolution methods for arbitrating the
meaning of something.foo, the net result is the same.

Current state of things in Scryer

.scryerrc

My .scryerrc module continues to grow to the point where I assume
it's going to include effectively the entire standard library.

It already currently has these:

:- use_module(library(lists)).
:- use_module(library(dcgs)).
:- use_module(library(reif)).
:- use_module(library(clpz)).
:- use_module(library(lambda)).
:- use_module(library(format)).
:- use_module(library(pairs)).
:- use_module(library(debug)).
:- use_module(library(time)).
:- use_module(library(files)).
:- use_module(library(os)).
:- use_module(library(iso_ext)).

And I'm sure that will continue to grow as time goes on.

I've noticed that I frequently forget in my examples to include
everything in my .scryerrc file -- this puts additional burden on the
reader. As someone who is also learning Scryer, I've noticed this same
practice in everything from posts to books on Prolog, you need to
sometimes need to dig significantly even through non-code sources to
find the definition of a predicate.

Qualifying modules only when necessary

I have noticed, probably in the name of readability, that module names
are typically elided except where technically necessary for the
compiler/interepter.

This will not work with readability at scale for reasons listed above.

Suggestions

This is not to say that things such as if_ or member should be
qualified as reif:if_ or lists:member, more likely they should just
be automatically available at runtime.

However, for instance, library(crypto) modules probably should be
namespace qualified, because they are unlikely to be commonly found in
code and out of context, someone might not know what module they come
from.

Within the application space, virtually all application code
(outside the current module) should be module qualified, and it would be
better if it could be path qualified from the project root or aliasable,
e.g, app.lib.foo:bar/2 or app/lib/foo:bar/2, whatever is most
appropriate for Prolog, or aliasable perhaps with
:- use_module_alias(app/lib/foo, lib/foo) or
:- use_module_alias(app/lib/foo, lib.foo) (whatever the case me be).

In general, we should be respectful of others reading our code and
module qualify things by default rather than only when technically
necessary. Code that is so common that it shouldn't be qualified should
be included automatically by default, because by convention that's what
we're all doing with our .scryerrc modules anyway and this will just
be one less hurdle for newcomers when they want to try our language.

ISO-compliance

Of course we want to stay isocompliant, so we need to make this an
extension rather than a change.

Recap

1. We need a way to alias modules via use_module/3 or something
   similar, i.e.: use_module(Module, Alias, Specifics)
2. We should get in the habit of qualifying everything except the
   builtins, and if we aren't doing that, we should probably start
   treating them as builtins unless a specific restrictive flag is
   present during startup

Please let me know what I'm overlooking, or how this is currently being
handled, criticims, and how it aligns with the goal of making Scryer
suitable for large scale projects.

[triska]

Of the exported library predicates, is there currently any overlap or ambiguity in names?

Also, if we consider the predicates exported by library(crypto) which you mention:

:- module(crypto,
          [hex_bytes/2,                  % ?Hex, ?Bytes
           crypto_n_random_bytes/2,      % +N, -Bytes
           crypto_data_hash/3,           % +Data, -Hash, +Options
           crypto_data_hkdf/4,           % +Data, +Length, -Bytes, +Options
           crypto_password_hash/2,       % +Password, ?Hash
           crypto_password_hash/3,       % +Password, -Hash, +Options
           crypto_data_encrypt/6,        % +PlainText, +Algorithm, +Key, +IV, -CipherText, +Options
           crypto_data_decrypt/6,        % +CipherText, +Algorithm, +Key, +IV, -PlainText, +Options
           ed25519_seed_keypair/2,       % +Seed, -KeyPair
           ed25519_new_keypair/1,        % -KeyPair
           ed25519_keypair_public_key/2, % +KeyPair, +PublicKey
           ed25519_sign/4,               % +KeyPair, +Data, -Signature, +Options
           ed25519_verify/4,             % +PublicKey, +Data, +Signature, +Options
           curve25519_generator/1,       % -Generator
           curve25519_scalar_mult/3,     % +Scalar, +Point, -Result
           crypto_name_curve/2,          % +Name, -Curve
           crypto_curve_order/2,         % +Curve, -Order
           crypto_curve_generator/2,     % +Curve, -Generator
           crypto_curve_scalar_mult/4    % +Curve, +Scalar, +Point, -Result
          ]).

Which of these predicates is most likely to benefit from module qualification because "someone might not know what module they come from"? Probably not the majority of these predicates, the 11 predicates which start with the prefix crypto_?

[jjtolton]

It's not because there is a conflict, it's because someone who is reading the code would want to know what library has the code in it if they are unsure.

I probably should've picked a better module. What I was trying to say is library(crypto) is unlikely to be a candidate for "should be loaded at runtime", whereas it seems pretty likely library(lists) and I would even argue library(clpz) seem like ones that should be always available, in 99% of situations there is no reason not to include them. So for instance, using member, if_, etc, are so common they should probably be available without any use_module. crypto probably would be pulled via use_module. If the module was written with the idea of module qualification, the crypto_ prefixes would've probably been elided in favor of crypto:curve_scalar_mult(Curve, Scalar, Point, Result), etc.

I am guessing the predicate names were most likely written so that there would be no need to module qualify them. However, I would still advocate that, for e.g., crypto:ed25519_keypair_public_key(Keypair, PublicKey) would be better than ed25519_keypair_public_key(Keypair, PublicKey), because someone who is new to the language would be able to track down the library better out of context.

Keep in mind that right now, this is probably not necessary. But in the future, when there is a package management system, environments, lots of users, this will probably become more important -- not necessarily for the library modules, BUT the standard library should set an example for the rest of packages to be used.

[triska]

from somelib import *

Note that there is no need to import all predicates from a module. For instance, you can use:

:- use_module(library(crypto), [hex_bytes/2]).

to import only hex_bytes/2 from library(crypto).

[triska]

I frequently forget in my examples to include everything in my .scryerrc file

When reporting an issue, I recommend to use the -f command line option of Scryer to ignore .scryerrc:

$ scryer-prolog -f

[hurufu]

Major Points

1. We need a way to alias modules via use_module/3 or something
   similar, i.e.: use_module(Module, Alias, Specifics)
2. We should get in the habit of qualifying everything except the
   builtins, and if we aren't doing that, we should probably start
   treating them as builtins unless a specific restrictive flag is
   present during startup

1. No
2. No

Adding explicit qualification to every predicate will incur non-trivial runtime overhead, and will make reasoning about terms much harder. Unification of terms would be nightmare. There are so many other reasons... I just don't have stamina to write all of them at once :)

[jjtolton]

1. No

No is not an argument silly.

2. No

This is how my nephew argues, too! 🤣

Adding explicit qualification to every predicate will incur non-trivial runtime overhead,

"Module qualification adds non-trivial runtime overhead?? In a way that can't be optimized by the implementation?? What OTHER problems does this language have??? Must not be suitable for serious work then." ~ anyone who hears that argument

and will make reasoning about terms much harder.

In what way? I think it would be much easier.

Unification of terms would be nightmare. There are so many other reasons... I just don't have stamina to write all of them at once :)

Unification of terms would be a nightmare... how, exactly? It seems like it would be easier.

[jjtolton]

The real zen of Prolog is internalizing that code is data, with explicit qualification this notion is destroyed.

Lisps such as Clojure are also homoiconic, and we heavily namespace qualify things. This doesn't destroy that in anyway. Can you explain how it destroys it in Prolog? I can point to several examples where module qualified predicates are processed as data.

[hurufu]

[...] Module qualification adds non-trivial runtime overhead??

Yes indeed, when you write for example maplist(module:pred, L) it will add term (:)/2 on the stack then it will add atom module then it will add atom pred and so on. Do you see the pattern? If you will do it for every predicate in a large program – it will cause an unnecessary performance overhead.

In a way that can't be optimized by the implementation??

Yes, in theory you can analyze the code and try to remove (:)/2 during goal expansion – but what's the point of adding them, to just what? remove?

Unification of terms would be a nightmare... how, exactly? It seems like it would be easier.

Just think about it, expression A = B is very simple, huh? But in the presence of modules it might fail if A = user:length(_,_) and B = lists:length(_,_) And if you will remove those qualifiers during compilation, then it will be even more unpredictable, sometimes A=B will succeed and sometimes it wouldn't...

Lisps such as Clojure are also homoiconic, and we heavily namespace qualify things. This doesn't destroy that in anyway. Can you explain how it destroys it in Prolog? I can point to several examples where module qualified predicates are processed as data.

As I said previously, suddenly you will have special cases in a lot of your predicates for unifying with (:)/2 and without and it, and it will be hard to make it performant, because you will always have a "defaulty" clauses:

pred(M:P) :- ... .
pred(P) :- P \= _:_, ... . 

I think you don't get semantics of (:)/2 fully, and I'm same as you, I'm still learning it! But for sure it isn't a "namespace" nor "fully qualified predicate name". For example ?- lists:call(length(A,B)). will succeed – it is more akin to switching context where predicate is invoked.

[hurufu]

Also consider a predicate mod:pred(G_1) which "outputs" a partial goal to be executed. How to call it?
call(mod:G_1, X) or call(G_1, X) or something else entirely? It has to be documented somehow.

Just try writing a program in your style, and you will see how unnecessary difficult it will become ;)

[hurufu]

GNU Prolog (a very respected Prolog implementation) doesn't support modules at all, and it was a deliberate choice.

[bakaq]

I've noticed that I frequently forget in my examples to include
everything in my .scryerrc file [...]

This is one of the reasons that my .scryerrc is empty, and that I always run public examples with the -f flag (which ignores the .scryerrc), so that everything is self-contained and reproducible.

Code that is so common that it shouldn't be qualified should
be included automatically by default, because by convention that's what
we're all doing with our .scryerrc modules anyway and this will just
be one less hurdle for newcomers when they want to try our language.

I think including extra predicates is an ISO conforming extension (not sure if strictly conforming though), but one of the things that I feel is very nice of Scryer Prolog is that it doesn't do that by default¹, so that it can be used to test if code is ISO compliant. I don't know if we want to put the strict ISO mode behind a flag, or if we should put the "nice" mode behind a flag instead, but I lean to the latter.

I feel a good way to do that would be to introduce a Scryer prologue (different but similar in concept to the Prolog prologue). This would include a lot of standard libraries that we use a lot in Scryer Prolog (like dgcs, reif, clpz, lists, lambda, pio, etc), and could be activated by a flag to the binary or (even better) a directive in the source code. We could give the same mechanism for the Prolog prologue, and make the Scryer prologue a superset of it as both of them evolve.

Footnotes

1. Mostly, error/2 exists. ↩

[UWN]

How are interfaces still maintained with explicit qualification in the source code? After all, any code can be reached with explicit qualification. Explicit qualification is needed at runtime in the Quintus/SICStus module system but only for these meta-arguments.

I'd go so far to claim that any explicit qualification (except those with multifile/1) should be avoided.

[jjtolton]

I like interfaces, interfaces are good. I am all for them. However, I would expect that module qualifying the interfaces would make them robust. What am I missing?

[UWN]

What am I missing?

Code that does not use explicit qualification does respect the interfaces without any further ado. So it is trivial for a programmer to tell how well-behaved the code is. This could be supported by a tool, but it's already fine without.

If there is explicit qualification everywhere, there is no way to tell, except by inspecting the interfaces. So the essential purpose of modules is taken away, and would need to be replaced by a new extra analysis (id est, a new module system on top). You can get that already without any module system.

[jjtolton]

I think this is a usage of "interfaces" with which I am unfamiliar, I assume we are not talking about the OO sort of interface? And it sounds like you are talking about something more specific than the concept of "how a programmer interacts with software"/"application programmer interface".

If you've explained this elsewhere such as on SO and you happen to have a link so I can read up on it I'd appreciate it. It sounds fascinating.

[jjtolton]

Code that does not use explicit qualification does respect the interfaces without any further ado. So it is trivial for a programmer to tell how well-behaved the code is. This could be supported by a tool, but it's already fine without.

If there is explicit qualification everywhere, there is no way to tell, except by inspecting the interfaces. So the essential purpose of modules is taken away, and would need to be replaced by a new extra analysis (id est, a new module system on top). You can get that already without any module system.

I know I must be misunderstanding because this is the opposite of OO interfaces, where the only way to know what the underlying execution is if you know the associated namespace of the invoked interface.

Aka something.append(x) would behave differently for something being a linked list, array, trie, hashset, etc. Usually there is an IAppend "interface" that defines the signature for the .append(x) method which is implemented concretely by the classes that support the IAppend interface.

But it sounds like this may be the inverse of that?

[triska]

The interface of a module are the predicates that it makes accessible without module qualification. These are the predicates that can safely be used. A module may in addition define predicates that are intended exclusively for use within that module itself. Separating the interface from the implementation is one of the main points, if not the main point, of a module system.

If you want to enforce explicit module qualification in all predicate calls, import every module without making any interface predicates available. For example:

:- use_module(library(crypto), []).

You then can still use the predicates in library(crypto), by using the module qualification crypto:..... However, in the same way, you can also (again) access predicates that are internal to library(crypto) and not part of its interface. Thus, explicit module qualification completely eliminates the distinction between interface predicates and internal predicates, which is a key attraction of a module system.

[jjtolton]

I am a bit surprised this is so contested.

Let's imagine a production file like this in the future:

:- use_module(library(lists)).
:- use_module(library(system)).
:- use_module(library(charsio)).
:- use_module(library(format)).
:- use_module(library(random)).
:- use_module(library(debug)).

:- use_module(app_utils).
:- use_module(data_access).
:- use_module(user_interface).
:- use_module(business_logic).
:- use_module(reporting).
:- use_module(authentication).
:- use_module(domain_models/user).
:- use_module(domain_models/product, [product/4]).
:- use_module(domain_models/order, [create_order/2]).
:- use_module(services/payment_gateway, [process_payment/3]). 
:- use_module(services/email_service).
:- use_module(domain_models/cart, [add_item/3, remove_item/3, get_total/1]).
:- use_module(domain_models/shipping, [calculate_shipping_cost/3]). 
:- use_module(domain_models/payment, [process_refund/2]).
:- use_module(services/notification_service, [send_order_confirmation/2]).
:- use_module(reporting/sales_reports, [generate_monthly_report/1, generate_yearly_report/1]).
:- use_module(reporting/customer_reports, [get_top_customers/1, analyze_purchase_history/2]).
:- use_module(tools/data_migration, [migrate_legacy_data/0]). 
:- use_module(tools/performance_testing, [run_load_test/2]).
:- use_module(ui_components/login_form).
:- use_module(ui_components/product_listing, [display_products/1]).
:- use_module(ui_components/shopping_cart, [show_cart/1, update_quantity/3]). 
:- use_module(admin_tools/user_management, 
              [add_user/3, remove_user/2, list_users/0]).
:- use_module(admin_tools/product_management, [add_new_product/4]).
:- use_module(integrations/thirdparty_api, [fetch_data/2]).

We currently have no (step-thru) debugging tools, no refactoring tools, no code exploration tools, nothing really except grep and your wits!

Let's consider the possibilities:

1. Everything is going to get named module_this, module_that by convention, like in C, for which we can already see evidence in library(queue), library(crypto) as mentioned above, library(tls), and library(sockets).
2. It's going to be extremely difficult to scale projects, as engineers could have to search through dozens of files to find a predicate definition.
3. The lack of ability to have aliases is going to encourage a very flat, very fat project structure and provide incentive for very large files because no one is going to want to have to prefix admin_tools_product_management__add_new_product/4 as a predicate name, or adm_tls_pd_mgmt__add_new_product/4.
4. There are plenty of predicates, where, for instance, parse_string might be a good name. library(ugraph) is a good example. neighbors/3, compose/3, before/2 are very common names.

Right now, even with the standard library, unless you happen to be very familiar with the entire standard library, it can be difficult to know where to look for something like length/2 and nth/3 most (new) people would expect in builtins, between/2 and between/3 are in different libraries, if_/3 most (new) people would expect to be in builtins, repeat/1 and repeat/0 are also in different libraries.

Now you might say, "pssssh, Jay, you're being silly. There aren't that many predicates in the standard library, and they are mostly well named, and most people aren't new for very long".

Well, you might be right, but I'm not really talking about the standard library, I'm talking about people (hopefully) copying the coding style of the standard library. I am talking about application code. Which I'm assuming we want, and we don't want Scryer to be considered a hobbyist language.

With a few people working on a project for 4-5 years ONLY, I have no doubt you could be looking a hundreds of files, 10s to 100s of thousands of lines of code, with a mix of standard library, 3rd party, and application files and packages. I'm honestly not sure how the idea of not module qualifying or module alias qualifying is justifiable.

Let's say someone gets hired to work on a Scryer Prolog project. It is 5 years old, has had 8 full time people and 3 contractors work on it, has 125,000 lines of code in it. There's no documentation or comments because that's how real code works and the documentation/comments that exist were written in year 2 and haven't been updated since. There's a half-assed test suite, most of which are commented out. You get hired because you're an expert in Prolog and "something's not working somewhere, but we're not sure what and we're not sure where, and it only happens sometimes".

If you can tell me with a straight face that you want to spend your time grepping predicate definitions, then you are a better poker player than I am! Either that or you would say, "I wouldn't take that job". But that's every job, everywhere.

[triska]

It's going to be extremely difficult to scale projects, as engineers could have to search through dozens of files to find a predicate definition.

Can the development environment take care of this? I expect a Prolog cross-referencer to find the answer within fractions of a second.

[hurufu]

Everything is going to get named module_this, module_that by convention, like in C

What's bad with that approach? I'm a C dev by trade, btw ;) Like seriously, what is the difference between module:predicate and module_predicate in your case when you insist on full qualification everywhere?

[hurufu]

Right now, even with the standard library, unless you happen to be very familiar with the entire standard library, it can be difficult to know where to look for something like length/2 and nth/3 most (new) people would expect in builtins, between/2 and between/3 are in different libraries, if_/3 most (new) people would expect to be in builtins, repeat/1 and repeat/0 are also in different libraries.

I see your pain here, I'm working on #2602, and I plan to add a warning that will suggest which module to use.

[jjtolton]

That's a GREAT point! Part of the whole reason I brought this up is because I'm studying the module system so I can write one for emacs. So far it does not seem like Scryer collects enough metadata to make this possible automatically. There's currently no way to associate predicates with modules because there is no way to enumerate modules, and no way to find the files associated with the Scryer modules without communicating with an external system, so we have some hills to climb before we get there.

Even with a cross-referencer, module qualification becomes immensely helpful. Autocomplete becomes much more useful, as when you begin the autocomplete sequence, it first attempts to match against a smaller number of modules, and then the remainder of the completion is constrained by the module. Excellent for discovery, and those tools can even automatically populate the use module directives!

However in the short term, no such tool exists. Prolog, like lisp, is also going to struggle a bit with this because of things like term and goal expansion.

But even languages that have sophisticated IDEs still typically qualify more often than not. I would suggest there should be a very good reason for not following this practice.

[jjtolton]

I'm a C dev by trade, btw ;)

Mad respect!

Like seriously, what is the difference between module:predicate and module_predicate in your case when you insist on full qualification everywhere?

I should point out that's why I'm advocating strongly for aliasing, not full qualification. Full qualification would be obnoxious and worse than what we have now!

With aliasing you have the option of shortening the qualification. It's also really helpful with tooling, autocomplete, refactoring, and just knowing what modules things come from.

But honestly I'm really surprised this is getting so much pushback, I actually kind of assumed everyone would be on board and we just hadn't gotten around to it yet!

It's certainly not a hill I feel the need to die on, I can write my own tools to do the job if needed.

I'm actually still not quite sure where this is coming from, other than I'm assuming Prolog traditionally has never had modules and so historically there has never been prefixing.

[triska]

Please let me know what I'm overlooking

One general comment regarding the number of predicates we expect: Due to a predicate being able to handle multiple modes, the number of library predicates that are needed to write interesting Prolog applications tends to be smaller than the number of, say, library functions in other languages. A predicate with N arguments can be invoked in 2^N different modes, so a single Prolog predicate with, say, 3 arguments would tend to be expressed with 8 functions in other programming languages.

This trend of order-of-magnitude reduction in library exports may be one of the reasons why this concern has so far not yet arisen.

[hurufu]

@jjtolton Do you remember that I called "programming in the large" a highly debated topic? #2591 (comment) Now you see ;)

[hurufu]

I think it is one of the reasons why we have so many different Prolog implementations.

[jjtolton]

@jjtolton Do you remember that I called "programming in the large" a highly debated topic? #2591 (comment) Now you see ;)

I see that now! 😅

I had no idea!!

[hurufu]

If you want more drama just read comp.lang.prolog usenet archive, sadly it isn't active anymore, there are like 1.5 people posting there.

[jjtolton]

The other thing I will say, most of the people on this board are so experienced I would never dream of telling any of you how to write code, especially Prolog.

I'm not talking about you guys, I'm talking about all the other people. The overworked engineer who hasn't slept much and has an impossible delivery schedule, and his beleaguered coworkers who need to review it. The new intern. The guy who's been doing javascript for 12 years and recently got a job here.

That's why I tagged this "in the large" and not "for people who are really, really good at it".

So if at any point you think I'm trying to tell you how you should write your code, I'm definitely not!!

But not everyone is you.

For example, I think Peter Norvig is one of the greatest Python developers there is, but his style should not be emulated. He can get away with it because he is Peter Norvig.

I think some of the people on this board are on par with him, in fact @triska is the only person I've ever seen to catch a mistake in his code!

So if you are already an expert, you of course can write however you like.

Imagine not what rules you would want to follow yourself, but what rules you would want the worst, most malevolent, lazy programmer you can imagine to follow. Imagine they wrote 18 months of code and then it was abandoned and handed over to you with zero explanation or documentation. What rules would you have wanted that person to follow?

[triska]

One point that may be relevant for these considerations: What does it mean now, in existing code, when we see an explicit module qualification?

Now, it means that something strange is going down, something to watch out for because it may have unintended and extreme effects. In existing code, explicit module qualification is a sign that something circumvents the agreed-upon library interface, or steps outside its own designated responsibilities, an overreach into other modules to access parts that they themselves expect to have total control over.

This is sometimes acceptable and currently even necessary. For instance, the well-known example of global goal expansions which need to be defined "into" the module user, as library(clpz) does it:

https://github.com/mthom/scryer-prolog/blob/9785a04b3a9a9b8a2fbf67d8b1012daa6c1cd408/src/lib/clpz.pl#L3158C1-L3159C13

An explicit module qualification is a sign that much care and attention is warranted in this part of the code, it currently serves as a warning that unusual things are occurring.

[hurufu]

May I rephrase: In the current Scryer module system using qualified predicates is like calling private methods from OO languages.

[jjtolton]

Just to stir the pot a little...

A few things to notice:

When I begin typing an alias, I am offered autocomplete options. When I confirm an alias and type /, the header is automatically updated appropriately so I do not have to break my flow. When I place the point after the / and prompt for completions, it kindly shows me what all of my options are for that namespace.

It's a really nice workflow.

As you can see, it's not much additional effort to include documentation, parameters information, cross-reference information, etc.

This really looks like it would be that horrible??

[UWN]

You are zooming onto one moment in program development. But this is not where most programs spend their existence. After having written the program, how can you now ensure that interfaces are respected? You need another tool for this.

And, similar tools are imaginable for other approaches that could be better than a mere grep(1) which is not that bad to start with.

[jjtolton]

Alright, I'm still not throwing in the towel on this. Behold.

On the left: an extensible framework for solving an A* problem.
Three panels to the right of that:
Modules implementing interfaces for actions//1, select//1, and handle//2.

The 4 panel display was chosen to highlight the symmetry in the interface design.

This has the highly desirable extensibility property that the code in the left panel, with the exception of line 66:

action_domains([bucket, move, rocks]).

does not need to be updated for new modules to participate in the successor/3 node predicate, nor do existing modules need to be updated with awareness of module accretions.

I do hope you would be able to look past the minor typographical nitpicks or API inefficiencies (it's pseudocode anyway) and consider the big picture:

1. The value this style of composition
2. What things could be done to support this style of composition.

Regarding the value of this style of composition, in my experience, it addresses the most common meta-problem faced across all industries. Obvious examples are in web development (where new APIs need to be added regularly), decision support (new reports are required), data processing (new pipelines need to be added), and automation (new workflows need to be added) -- you are routinely required to add new functionality without introducing regressions.

Regarding support, as mentioned above there is currently no way to iterate over modules loaded into Scryer and that's a pretty big weakness in this design, as it requires manually updating at least one fact per new domain added.

Also this is not immune to typos that result in a silent false/0 -- these typos have been the scourge of my existence. I introduced a design using facts called attr which would by used be set/3 and get/2 predicates along with Module/Functor qualified functors that would be checked using the Module:hasattr/1 predicate -- this would preemptively prevent predicates from failing silently because of functor predicate signature incompatibilities introduced by mistyping a functor.

It is pretty unfortunate that the code forhasattr/1 needs to be duplicated across all modules. Maybe this could be solved with term expansion?

Additionally, the actions//1, select//1, and handle//2 schema is entirely implicit and out-of-band, there is no "meta" technique to enforce that a new module has the required predicates or that they meet any of the guarantees of the calling predicate.

Clearly it would be much better if we could borrow concepts from Algebraic Data Types and Structured Programming (but handled in a Prolog style) to address some of these issues.

Finally, I hope it is obvious that module system has a lot of potential. You may not like this style of composition(, or the names that were selected, or my IDE color schemes, or ...) -- but if you don't, I hope you have a much better answer for the problem this style of composition solves for!

[hurufu]

Interesting, I like how you call same predicate across different modules (aka domains). I just want to emphasize that there is nothing special about ((:)/2) and you can store your domains using any other term, and convert it to Module:Predicate just before calling it. Also I'm still not convinced that qualification is good for every or even some predicates, but for sure when it is needed – it is a unavoidable.

As a personal story, on a few occasions I used a technique that every business rule in a system was a single predicate. So for example instead of writing speed(robot, 100). I would write: rule(speed, robot, 100). It also proved useful, because I could emulate higher order reasoning, so instead of Predicate(_, 100) which in incorrect syntactically, I could just do rule(Predicate, _, 100).

[hurufu]

After second thought I think it can be done without explicit qualification, using multifile predicates: :- multifile(select/1). But I don't know how yet...

[jjtolton]

So it turns out goal_expansion/2 and term_expansion/2 solve every single one of my problems and way, way more on top of that!

This was the slide I gave about the problems all programming languages face:

And this is the one about how they are solved in Scryer:

Excited to get the rest of the slides together and demonstrate the techniques I'm using to achieve the best possible extensibility available in any language.