Simplify the build process using git submodules

[rwhitby]

I think the build process could be simplified by using git submodules:

1. sudo apt install libftdi1-dev libudev-dev git cmake build-essential tclsh clang tcl-dev libreadline-dev flex bison python3-dev libboost-all-dev libqt5-base-dev-tools libeigen3-dev python3 python3-pip python3-yaml pypy3 pkg-config libqt5opengl5-dev
2. clone/build/install yosys from https://github.com/YosysHQ/yosys or download a release from https://github.com/YosysHQ/oss-cad-suite-build/releases note: test have been performed with Yosys 0.13+28 (git sha1 bf85dfee5, gcc 10.2.1-6 -fPIC -Os)
3. git clone https://github.com/kintex-chatter/xc7k325t-blinky-nextpnr.git

This would change to include --recurse-submodules

4. git clone --recurse-submodules https://github.com/kintex-chatter/nextpnr-xilinx.git

This could be made a submodule of xc7k325t-blinky-nextpnr.git and then the branch and specific commit could be controlled. that would remove this step. The nextpnr-xilinx repository could also submodule the kintex7-specific prjxray-db repository.

5. cd nextpnr-xilinx

This would change to cd xc7k325t-blinky-nextpnr/nextpnr-xilinx

6. git checkout xilinx-upstream

This would no longer be required

7. mkdir build
8. pushd build
9. cmake -DARCH=xilinx -DCMAKE_INSTALL_PREFIX=~/opt/nextpnr ..
10. make -j2 && make install
11. popd
12. pushd xilinx/external

This would no longer be required

13. rm -rf prjxray-db

This would no longer be required

14. git clone -b k325 https://github.com/kintex-chatter/prjxray-db.git

This would no longer be required

15. popd

This would no longer be required

16. python3 xilinx/python/bbaexport.py --device xc7k325tffg676-1 --bba xilinx/xc7k325tffg676-1.bba
17. build/bbasm -l xilinx/xc7k325tffg676-1.bba xilinx/xc7k325tffg676-1.bin
18. mkdir -p ~/opt/nextpnr/xilinx-chipdb
19. ln -s $PWD/xilinx/external/prjxray-db ~/opt/nextpnr/
20. cp xilinx/xc7k325tffg676-1.bin ~/opt/nextpnr/xilinx-chipdb/

[rwhitby]

I will test this theory out, and if it works I'll submit a pull request.

I'm testing on Intel Ubuntu 20.04.4 LTS native and Ubuntu 20.04.4 LTS in WSL2 on Windows.

[rwhitby]

First pull request (for adding prjxray.db into nextpnr-xilinx) is at openXC7/nextpnr-xilinx#1 which removes the need for steps 12 through 15.

Once this one is merged, I'll submit a second pull request to add nextpnr-xilinx to the current repo and remove steps 4 through 6.

[unbtorsten]

Thank you, for the suggestions @rwhitby! I agree that this will simplify development.

In the long run, our focus should be an update of upstream repositories, and a build procedure that clones directly from prjxray. This will only take an adjustment of the submodule. I opened openXC7/nextpnr-xilinx#2 to remind us of this, whenever upstream prjxray-db is updated correspondingly.

Edit: Update of upstream is pending, cf. f4pga/prjxray#1794 and f4pga/prjxray#1844

[unbtorsten]

@rwhitby: I have updated the README correspondingly. I'd say we're ready for the next step of adding kintex-chatter/nextpnr-xilinx.git as submodule.

[rwhitby]

@unbtorsten Thanks for the branch fix. Second step is coming soon.

[rwhitby]

Should we add https://github.com/kintex-chatter/prjxray.git as a submodule too?

Then it will be in a known place for the Makefile to reference.

[rwhitby]

I also think that the top-level Makefile could automatically run most of these steps with a "setup" target.

The following seems to work on Intel Ubuntu 20.04.4 LTS and Intel macOS Montery 12.1:

rwhitby#1 (this depends on #20 being merged first, so has not yet been submitted here to prevent confusion).

The instructions would then simplify to:

1. git clone --recurse https://github.com/kintex-chatter/xc7k325t-blinky-nextpnr.git
2. cd xc7k325t-blinky-nextpnr
3. make setup
4. make BOARD=qmtech

[unbtorsten]

Should we add https://github.com/kintex-chatter/prjxray.git as a submodule too?

Then it will be in a known place for the Makefile to reference.

If you feel like automating this, I would opt for making this optional (if declined, tell the user to set path to prjxray manually). What do you think?

[unbtorsten]

I also think that the top-level Makefile could automatically run most of these steps with a "setup" target.

rwhitby#1 (this depends on #20 being merged first, so has not yet been submitted here to prevent confusion).

I added some comments in the above mentioned pull request for your fork to continue the discussion there.

[jrrk2]

I think yosys should also a a sub-repo since most distributions will not have a sufficiently up-to-date version available as a binary dependency

[rwhitby]

I think yosys should also a a sub-repo since most distributions will not have a sufficiently up-to-date version available as a binary dependency

As no changes are made to yosys, I found it easiest just to install oss-cad-suite binary nightly and put it in the path.

[unbtorsten]

I notice that we are mixing system requirements, make setup, and make blinky, which is where things can get confusing.
I propose that we separate the build environment and tools that are the same for all Kintex7 packages from the data that is specific to the Kintex7 packages:

1. List of system requirements (available from upstream) to be on a given PATH
   • yosys
     Steps to clone/build/install: download sources from https://github.com/YosysHQ/yosys or a release from https://github.com/YosysHQ/oss-cad-suite-build/releases
     note: test have been performed with Yosys 0.13+28 (git sha1 bf85dfee5, gcc 10.2.1-6 -fPIC -Os)
   • prjxray
     prjxray is provided as a submodule until its installation process is refactored, cf. Relationship to prjxray's fasm2frames chipsalliance/f4pga-xc-fasm#19. Alternatively, you may set XRAY_DIR to the path where Project Xray has been cloned and built, see https://symbiflow.readthedocs.io/en/latest/prjxray/docs/db_dev_process/readme.html#quickstart-guide for details. You will need to follow steps 2-5. You may be able to skip installation of Vivado if you do not plan to run the fuzzers.
   • openFPGALoader
     Clone/build/install from https://github.com/trabucayre/openFPGALoader
     • for xc7k325tffg676-1, cf. Xilinx XC7K325T support trabucayre/openFPGALoader#178
     • for xc7k325tffg900-2, cf. Add board and cable defaults for genesys2 trabucayre/openFPGALoader#180
2. Adaptation of build environment (independent of target Kintex package)
   • prjxray-db::k325 (update of prjxray pending)
3. Custom builds that depend on target Kintex package
   • nextpnr-xilinx
     • prjxray-db::k325 customization: Contribute xc7k update to upstream nextpnr-xilinx#3
     • Bugfix: Adapt to updated prjxray filestructure gatecat/nextpnr-xilinx#33
   • constraints
   • blinky.v

Feel free to edit this comment to rearrange items between lists so that we can reach consensus.

I think the goal for the 'tools' should be to send pull requests to upstream repositories, so that make setup would automate clone/build/install of standard packages. See SymbiFlow/yosys#96 for an example at prjxray previously depending on experimental changes to yosys. On the other hand, make blinky could contribute to the corresponding documentation in the long run.

[jrrk2]

If we want to be portable to Mac-M1, I don’t think we can rely on oss-cad-suite.

[unbtorsten]

I think we should focus on and clone/build only those repositories that require changes in comparison to their respective upstream. Creating a universal FOSS build environment is the objective of Symbiflow et al.

That said, I have updated the proposal for a scope of README and Makefile(s) stated in #18 (comment). My distribution, too, ships a version of yosys that is outdated with regard to our needs. Nevertheless, the required source and binaries are available from upstream. The same applies for prjxray. It is thus sufficient to rely on them being installed (in specific paths) as opposed to cloning them. The current revision of the README process reflects this and calls for installation of specific files in steps 0, 1, and 14.

[rwhitby]

I agree that anything upstream that is unchanged, should just be installed/used from upstream, and pull requests sent as soon as possible to incorporate any changes to upstream (as I have done already for openFPGALoader).

[rwhitby]

Pull request #25 covers the second part of my proposal.