Consistency for repo names and makefile locations.

[Earnestly]

Edit: I'm not sure where this should be posted but as this repo could benefit
from these changes, it seemed the most appropriate.

In order to make build systems and management more straight forward I would
like to depend on the internal consistency of repo names and makefiles.

For example, each repo could be named after the object it builds without the
_libretro.{so,dll} prefix.

This allows us to assume what what the core's name will be based purely on the
repo name itself.

Another change would be to make the builds consistent. For example it would be
nice if we could assume that make -f Makefile.libretro would always work
regardless.

Some repos seem to contain libretro specific builds in the libretro/
directory. This is also fine, but it would be nicer if this was consistent across
all of them.

Overall these two properties would allow for a nice way to manage cores without
needing to maintain whitelists or blacklists.

Here's a simple usecase as a result of these changes:

core_directory="${XDG_DATA_HOME:-$HOME/.local/share}"/retroarch/cores

# Note: genesis_plus_gx won't work as the repo name differs. :-(
cores=(pcsx_libretro mgba nestopia genesis_plus_gx)

confirm() {
    local -l reply
    read -p "Rebuild? [Y/n] " reply
    [[ -z "$reply" || "$reply" = y?(es) ]]
}

# XXX This assumes it is run in the build directory, a proper script would
#     account for this.
for core in "${cores[@]}"; do
    if [[ ! -d "$core" ]]; then
        git clone --depth 1 git://github.com/libretro/"$core"
    fi

    pushd "$core"

    git fetch

    if [[ "$(git rev-parse HEAD)" != "$(git rev-parse @{upstream})" ]]; then
        git log --oneline --stat ..@{upstream}
        if confirm; then
            make -f Makefile.libretro
            mv -f -- "$core"_libretro.so "$core_directory"
        fi
    fi  

    popd
done

[Earnestly]

Some findings so far:

Issue                      With what                     What's wrong

Repo name:         snes9x-next                => snes9x_next_libretro.so
Repo name:         Genesis-Plus-GX            => genesis_plus_gx_libretro.so
Makefile location: nestopia/Makefile.libretro => nestopia/libretro/Makefile
Makefile location: libretro-ppsspp            => libretro-ppsspp/libretro/Makefile
Repo name:         libretro-ppsspp            => ppsspp_libretro.so
Makefile naming:   libretro-fba               => makefile.libretro
Repo name:         libretro-fba               => fba_alpha_libretro.so

Personal opinion:

• Repo names should reflect the name of the built core without the _libretro.so
  or _libretro.dll suffix.

  For example:

  Genesis-Plus-GX => genesis_plus_gx
  libretro-fba    => fba_alpha
  

• Makefiles should be stored in a directory named libretro/ and called either
  Makefile or makefile (as gmake will look for either, Makefile would be
  more portable though.)

[andres-asm]

We don't really "own" all the repos, so it's not entirely possible.
The libretro-super script already allows building individual cores (when updates are found) so I think you could leverage that

./libretro-super nestopia should build nestopia for instance

[Earnestly]

The approach is not tenable for anything large scale or remotely automatic
because it will require constant maintenance and guess work. Compare the
size of the script to a simple loop.

A consistent scheme would expose a much nicer "API" to consumers of the
git repos without much effort and without any extra complexity.

[andres-asm]

It's easier to keep in-line one script than 30 or so repositories

[Earnestly]

But it's not simpler and will make life harder for others for almost no benefit.

I really don't understand your point here. Yes, doing nothing now will be
much easier than fixing it.

As to your point about ownership; that's fine. Inconsistency like that can be
mostly worked around automatically using any plain old associative data
structure. It's also something you can't fix, I can concede that point, but
the location of makefiles seems fixable...

[andres-asm]

Doing nothing? have you seen the commit log?
I just think getting OCD about the different naming schemes in various independent repositories is a waste of time. But it's not my decision to make I was just mentioning my POV on the matter.

[Earnestly]

So you don't see that having to special case every single repository instead of
a single unified interface as a problem?

I don't really know what to say to that, other than yes, let's see what others
in charge have to say. Either way I would like to know if this is something I
need to deal with before wasting my time on it.

[andres-asm]

The whole idea is that cores are independent from RA and from the libretro organization.
Many cores are maintained by libretro of course but as the API gains traction more people will adopt it.

Example mgba. He doesn't even use a static makefile, and we can't force him of course.

Anyway I see your point, but if that structural change happened I would have to update 30, or 60 or so recipes to make sure everything was working again. I'm against this unless someone wants to make sure the buildbot can still build everything on the 7 platforms it's doing now.

[Earnestly]

It would be nice if at least libretro required some kind of consistency when it
comes to the build. mgba is consistent with most of the projects, by using
Makefile.libretro.

Without rules about these things developers could do whatever they wanted
making life harder for others without realising it.

I don't want to follow the route Window's took and attempt to special case
every single problem software, e.g. hacking the memory allocator to prevent
certain software from crashing with use-after-free bugs.

So far most repositories tend to use either Makefile.libretro or
makefile.libretro or libretro/Makefile which is tolerable to hack around,
however this issue isn't about hacking around but trying to see if there's any
interest in making it more user friendly (consistency is key for this).

If we can get a nice consistent location for these things then the build systems
around RA cores will be greatly simplified.

A more extreme solution would be to provide some kind of API but I think that's
totally unnecessary when we could just reuse the existing infrastructure without
much ceremony.

Edit: I think your comment about having to fix upwards of 60 recipes should be
all we need to know about the current modus operandi.

# Seriously?
    if [[ -e Makefile.libretro ]]; then
        make -f Makefile.libretro
    elif [[ -e makefile.libretro ]]; then
        make -f makefile.libretro
    elif [[ -d libretro ]]; then
        make -C libretro
        core="libretro/$core"
    fi

    if [[ -e "$core" ]]; then
        mv -f -- "$core" "$core_directory"
    fi

[andres-asm]

So we should tell endrift, hey you should use a static Makefile if you want to make a libretro port? That won't work, and that's why Twinaphex forked it and added a static Makefile that needs to be updated whenever upstream changes happen. I still do my personal builds with cmake, since that "just works"

Same thing with PPSSPP, forcing those changes will achieve nothing for some platforms. We'll still have to use MSVC for windows for instance. And who knows for iOS.

As I said, I'm against this, not because of laziness, because I'd rather spend my time doing something else like I've been doing this week, so yeah changes that affect how builds are done even if it's for the greater good should imply a commitment of the person who does such changes to make sure that the systems that use the current build systems still work after that.

[Earnestly]

And yet ppsspp is fine with libretro/Makefile?
And yet mgba is fine with Makefile.libretro?

I don't understand what you're saying. What would be so terrible if both used Makefile.libreto or libretro/Makefile?

Do we really need to even differentiate between Makefile.libretro and makefile.libretro?

[andres-asm]

ppsspp is fine for linux with libretro/Makefile
mgba is fine for linux with Makefile.libretro

It's not terrible, but as I said, changing all the repos to be like that would be a huge task to undertake, and if it was gonna happen then again I'd expect commitment from the people who make such changes to keep the current systems working.

[andres-asm]

BTW, I don't mean to be rude, again it's just my opinion. We've had these before, people come propose changes, and then leave at the middle of the implementation, and then someone has to take over and make stuff work.

[Earnestly]

Eugh. This issue is then far too late. Nevermind.

[inactive123]

We'll look at this after 1.1 is out.

If that isn't fast enough, then I'd really appreciate if somebody other than me puts the work in when it comes to these consistency changes. I can't be expected to do it all.

[knghtbrd]

Some of these problems will be obsolete with the new libretro-super.sh (from scratch) which would be uploaded by now if not for getting called away for a family emergency. It wasn't hard to do since I'd already basically rewritten it from scratch in the past two months anyway, but it's cleaner now.

In the day or so I had to work on it, it already does by default:

• Check the hash
• Pull changes (--fetch)
• Check the hash again
• See if the working directory is dirty
• Stop there if the WD is clean and the hashes are the same
• Clean the core (--clean)
• Compile the core (--compile, with --build as alias for --clean --compile)

It can also zip cores (--package) for upload to the buildbot server, but as yet lacks an upload facility rule. It does know which cores to build on which systems (although I'm not sure cap32 is supposed to build on a Mac, it doesn't…) It's got a multi-target make for bsneses and mame, mame works with it, and the only cores it can't handle are the fba subcores for consoles which are easily added at this point.

I'll need a few hours with it before it's ready for pushing Wednesday because it needs to stop using libretro-config.sh because the end result will be able to build cores for more than one arch/platform, if you have the ability to do that (32/64 bit on Windows or Mac, iOS, etc.)

Did I mention that the code is actually simpler, easy to follow, commented, and the main script is under 700 lines when I last looked at it? :)

[Earnestly]

I'd probably stick with a simpler personal solution (being personal means I can
get away with more assumptions and support less requirements). Overall 71 lines
of (idiomatic) bash instead of 700 :-P

I'm sure the new script will be super handy for many people, but I mainly asked
this here to try and make it simpler for everyone as well who want to go their
own route.

Ultimately the repos aren't too bad, there are a few standards which make
building for Linux only pretty straight forward.

Edit: Not spectacular, but my simple script is here