Hosted resources search

.github/workflows/pr-testing.yml

@@ -0,0 +1,33 @@
+name: PR testing
+
+on:
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  mavenTesting:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Cache local Maven repository
+      uses: actions/cache@v2
+      with:
+        path: ~/.m2/repository
+        key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+        restore-keys: |
+          ${{ runner.os }}-maven-
+    - name: Set up the Java JDK
+      uses: actions/setup-java@v2
+      with:
+        java-version: '11'
+        distribution: 'adopt'
+
+    - name: Run all tests
+      run: |
+        mvn -B clean test -DskipTests=false --file pom.xml
+        if [ -f "target/site/jacoco/index.html" ]; then echo "Total coverage: $(cat target/site/jacoco/index.html | grep -o 'Total[^%]*%' | grep -o '[0-9]*%')"; fi
+
+    - name: Log coverage percentage
+      run: |
+        if [ ! -f "target/site/jacoco/index.html" ]; then echo "No coverage information available"; fi
+        if [ -f "target/site/jacoco/index.html" ]; then echo "Total coverage: $(cat target/site/jacoco/index.html | grep -o 'Total[^%]*%' | grep -o '[0-9]*%')"; fi

.gitignore

@@ -1,6 +1,8 @@
 /db*
+/lists/
 /bin/
 /target/
+/qortal-backup/
 /log.txt.*
 /arbitrary*
 /Qortal-BTC*
@@ -14,5 +16,18 @@
 /settings.json
 /testnet*
 /settings*.json
-/testchain.json
-/run-testnet.sh
+/testchain*.json
+/run-testnet*.sh
+/.idea
+/qortal.iml
+.DS_Store
+/src/main/resources/resources
+/*.jar
+/run.pid
+/run.log
+/WindowsInstaller/Install Files/qortal.jar
+/*.7z
+/tmp
+/data*
+/src/test/resources/arbitrary/*/.qortal/cache
+apikey.txt

AutoUpdates.md

@@ -1,5 +1,20 @@
 # Auto Updates
 
+## TL;DR: how-to
+
+* Prepare new release version (see way below for details)
+* Assuming you are in git 'master' branch, at HEAD
+* Shutdown local node if running
+* Build auto-update download: `tools/build-auto-update.sh` - uploads auto-update file into new git branch
+* Restart local node
+* Publish auto-update transaction using *private key* for **non-admin** member of "dev" group:
+	`tools/publish-auto-update.sh non-admin-dev-member-private-key-in-base58`
+* Wait for auto-update `ARBITRARY` transaction to be confirmed into a block
+* Have "dev" group admins 'approve' auto-update using `tools/approve-auto-update.sh`
+	This tool will prompt for *private key* of **admin** of "dev" group
+* A minimum number of admins are required for approval, and a minimum number of blocks must pass also.
+* Nodes will start to download, and apply, the update over the next 20 minutes or so (see CHECK_INTERVAL in AutoUpdate.java)
+
 ## Theory
 * Using a specific git commit (e.g. abcdef123) we produce a determinstic JAR with consistent hash.
 * To avoid issues with over-eager anti-virus / firewalls we obfuscate JAR using very simplistic XOR-based method.
@@ -25,8 +40,8 @@ The same method is used to obfuscate and de-obfuscate:
 
 ## Typical download locations
 The git SHA1 commit hash is used to replace `%s` in various download locations, e.g.:
-* https://github.com/QORT/qortal/raw/%s/qortal.update
-* https://[email protected]/QORT/qortal/%s/qortal.update
+* https://github.com/Qortal/qortal/raw/%s/qortal.update
+* https://[email protected]/Qortal/qortal/%s/qortal.update
 
 These locations are part of the org.qortal.settings.Settings class and can be overriden in settings.json like:
 ```
@@ -45,4 +60,12 @@ $ java -cp qortal.jar org.qortal.XorUpdate
 usage: XorUpdate <input-file> <output-file>
 $ java -cp qortal.jar org.qortal.XorUpdate qortal.jar qortal.update
 $
-```
+```
+
+## Preparing new release version
+
+* Shutdown local node
+* Modify `pom.xml` and increase version inside `<version>` tag
+* Commit new `pom.xml` and push to github, e.g. `git commit -m 'Bumped to v1.4.2' -- pom.xml; git push`
+* Tag this new commit with same version: `git tag v1.4.2`
+* Push tag up to github: `git push origin v1.4.2`

DATABASE.md

@@ -4,10 +4,10 @@ You can examine your node's database using [HSQLDB's "sqltool"](http://www.hsqld
 It's a good idea to install "rlwrap" (ReadLine wrapper) too as sqltool doesn't support command history/editing.
 
 Typical command line for sqltool would be:
-`rlwrap java -cp ${HSQLDB_JAR}:${SQLTOOL_JAR} org.hsqldb.cmdline.SqlTool --rcFile=${SQLTOOL_RC} qora`
+`rlwrap java -cp ${HSQLDB_JAR}:${SQLTOOL_JAR} org.hsqldb.cmdline.SqlTool --rcFile=${SQLTOOL_RC} qortal`
 
 `${HSQLDB_JAR}` should be set with pathname where Maven downloaded hsqldb, 
-typically `${HOME}/.m2/repository/org/hsqldb/hsqldb/2.5.0/hsqldb-2.5.0.jar`
+typically `${HOME}/.m2/repository/org/hsqldb/hsqldb/2.5.1/hsqldb-2.5.1.jar`
 
 `${SQLTOOL_JAR}` should be set with pathname where Maven downloaded sqltool,
 typically  `${HOME}/.m2/repository/org/hsqldb/sqltool/2.5.0/sqltool-2.5.0.jar`
@@ -25,10 +25,16 @@ Above `url` component `file:db/blockchain` assumes you will call `sqltool` from
 
 Another idea is to assign a shell alias in your `.bashrc` like:
 ```
-export HSQLDB_JAR=${HOME}/.m2/repository/org/hsqldb/hsqldb/2.5.0/hsqldb-2.5.0.jar
+export HSQLDB_JAR=${HOME}/.m2/repository/org/hsqldb/hsqldb/2.5.1/hsqldb-2.5.1.jar
 export SQLTOOL_JAR=${HOME}/.m2/repository/org/hsqldb/sqltool/2.5.0/sqltool-2.5.0.jar
 alias sqltool='rlwrap java -cp ${HSQLDB_JAR}:${SQLTOOL_JAR} org.hsqldb.cmdline.SqlTool --rcFile=${SQLTOOL_RC}'
 ```
 So you can simply type: `sqltool qortal`
 
 Don't forget to use `SHUTDOWN;` before exiting sqltool so that database files are closed cleanly.
+
+## Quick and dirty version
+
+With `sqltool-2.5.0.jar` and `qortal.jar` in current directory, and database in `db/`
+
+`java -cp qortal.jar:sqltool-2.5.0.jar org.hsqldb.cmdline.SqlTool --inlineRc=url=jdbc:hsqldb:file:db/blockchain`

Dockerfile

@@ -0,0 +1,26 @@
+FROM maven:3-openjdk-11 as builder
+
+WORKDIR /work
+COPY ./ /work/
+RUN mvn clean package
+
+###
+FROM openjdk:11
+
+RUN useradd -r -u 1000 -g users qortal && \
+    mkdir /usr/local/qortal /qortal && \
+    chown 1000:100 /qortal
+
+COPY --from=builder /work/log4j2.properties /usr/local/qortal/
+COPY --from=builder /work/target/qortal*.jar /usr/local/qortal/qortal.jar
+
+USER 1000:100
+
+EXPOSE 12391 12392
+HEALTHCHECK --start-period=5m CMD curl -sf http://127.0.0.1:12391/admin/info || exit 1
+
+WORKDIR /qortal
+VOLUME /qortal
+
+ENTRYPOINT ["java"]
+CMD ["-Djava.net.preferIPv4Stack=false", "-jar", "/usr/local/qortal/qortal.jar"]

README.md

@@ -9,4 +9,4 @@
 - Create basic *settings.json* file: `echo '{}' > settings.json`
 - Run JAR in same working directory as *settings.json*: `java -jar target/qortal-1.0.jar`
 - Wrap in shell script, add JVM flags, redirection, backgrounding, etc. as necessary.
-- Or use supplied example shell script: *run.sh*
+- Or use supplied example shell script: *start.sh*

TestNets.md

@@ -0,0 +1,69 @@
+# How to build a testnet
+
+## Create testnet blockchain config
+
+- You can begin by copying the mainnet blockchain config `src/main/resources/blockchain.json`
+- Insert `"isTestChain": true,` after the opening `{`
+- Modify testnet genesis block
+
+### Testnet genesis block
+
+- Set `timestamp` to a nearby future value, e.g. 15 mins from 'now'
+	This is to give yourself enough time to set up other testnet nodes
+- Retain the initial `ISSUE_ASSET` transactions!
+- Add `ACCOUNT_FLAGS` transactions with `"andMask": -1, "orMask": 1, "xorMask": 0` to create founders
+- Add at least one `REWARD_SHARE` transaction otherwise no-one can mint initial blocks!
+	You will need to calculate `rewardSharePublicKey` (and private key),
+	or make a new account on mainnet and use self-share key values
+- Add `ACCOUNT_LEVEL` transactions to set initial level of accounts as needed
+- Add `GENESIS` transactions to add QORT/LEGACY_QORA funds to accounts as needed
+
+## Testnet `settings.json`
+
+- Create a new `settings-test.json`
+- Make sure to add `"isTestNet": true,`
+- Make sure to reference testnet blockchain config file: `"blockchainConfig": "testchain.json",`
+- It is a good idea to use a separate database: `"repositoryPath": "db-testnet",`
+- You might also need to add `"bitcoinNet": "TEST3",` and `"litecoinNet": "TEST3",`
+
+## Other nodes
+
+- Copy `testchain.json` and `settings-test.json` to other nodes
+- Alternatively, you can run multiple nodes on the same machine by:
+	* Copying `settings-test.json` to `settings-test-1.json`
+	* Configure different `repositoryPath`
+	* Configure use of different ports:
+		+ `"apiPort": 22391,`
+		+ `"listenPort": 22392,`
+
+## Starting-up
+
+- Start up at least as many nodes as `minBlockchainPeers` (or adjust this value instead)
+- Probably best to perform API call `DELETE /peers/known`
+- Add other nodes via API call `POST /peers <peer-hostname-or-IP>`
+- Add minting private key to node(s) via API call `POST /admin/mintingaccounts <minting-private-key>`
+	This key must have corresponding `REWARD_SHARE` transaction in testnet genesis block
+- Wait for genesis block timestamp to pass
+- A node should mint block 2 approximately 60 seconds after genesis block timestamp
+- Other testnet nodes will sync *as long as there is at least `minBlockchainPeers` peers with an "up-to-date" chain`
+- You can also use API call `POST /admin/forcesync <connected-peer-IP-and-port>` on stuck nodes
+
+## Dealing with stuck chain
+
+Maybe your nodes have been offline and no-one has minted a recent testnet block.
+Your options are:
+
+- Start a new testnet from scratch
+- Fire up your testnet node(s)
+- Force one of your nodes to mint by:
+	+ Set a debugger breakpoint on Settings.getMinBlockchainPeers()
+	+ When breakpoint is hit, change `this.minBlockchainPeers` to zero, then continue
+- Once one of your nodes has minted blocks up to 'now', you can use "forcesync" on the other nodes
+
+## Tools
+
+- `qort` tool, but use `-t` option for default testnet API port (62391)
+- `qort` tool, but first set shell variable: `export BASE_URL=some-node-hostname-or-ip:port`
+- `qort` tool, but prepend with one-time shell variable: `BASE_URL=some-node-hostname-or-ip:port qort ......`
+- `peer-heights`, but use `-t` option, or `BASE_URL` shell variable as above
+

WindowsInstaller/Install Files/AppData/settings.json

@@ -0,0 +1,3 @@
+{
+  "apiDocumentationEnabled": true
+}

WindowsInstaller/Install Files/log4j2.properties

@@ -0,0 +1,70 @@
+rootLogger.level = info
+# On Windows, uncomment next line to set dirname:
+# property.dirname = ${sys:user.home}\\AppData\\Local\\qortal\\
+property.filename = ${sys:log4j2.filenameTemplate:-log.txt}
+
+rootLogger.appenderRef.console.ref = stdout
+rootLogger.appenderRef.rolling.ref = FILE
+
+# Suppress extraneous bitcoinj library output
+logger.bitcoinj.name = org.bitcoinj
+logger.bitcoinj.level = error
+
+# Override HSQLDB logging level to "warn" as too much is logged at "info"
+logger.hsqldb.name = hsqldb.db
+logger.hsqldb.level = warn
+
+# Support optional, per-session HSQLDB debugging
+logger.hsqldbRepository.name = org.qortal.repository.hsqldb
+logger.hsqldbRepository.level = debug
+
+# Suppress extraneous Jersey warning
+logger.jerseyInject.name = org.glassfish.jersey.internal.inject.Providers
+logger.jerseyInject.level = off
+
+# Suppress extraneous Jersey EOF 'errors' (actually remote peers disconnecting early)
+logger.jerseyEOF.name = org.glassfish.jersey.server.internal
+logger.jerseyEOF.level = off
+
+# Suppress extraneous Jetty entries
+# 2019-02-14 11:46:27 INFO  ContextHandler:851 - Started o.e.j.s.ServletContextHandler@6949e948{/,null,AVAILABLE}
+# 2019-02-14 11:46:27 INFO  AbstractConnector:289 - Started ServerConnector@50ad322b{HTTP/1.1,[http/1.1]}{0.0.0.0:9085}
+# 2019-02-14 11:46:27 INFO  Server:374 - jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_181-b13
+# 2019-02-14 11:46:27 INFO  Server:411 - Started @2539ms
+logger.jetty.name = org.eclipse.jetty
+logger.jetty.level = warn
+# Even more extraneous Jetty output
+# 2019-01-26 02:18:10 WARN  ResourceService:718 - java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
+logger.jettyRS.name = org.eclipse.jetty.server.ResourceService
+logger.jettyRS.level = error
+
+# Suppress extraneous slf4j entries
+# 2019-02-14 11:46:27 INFO  log:193 - Logging initialized @1636ms to org.eclipse.jetty.util.log.Slf4jLog
+logger.slf4j.name = org.slf4j
+logger.slf4j.level = warn
+
+# Suppress extraneous Reflections entry
+# 2019-02-27 10:45:25 WARN  Reflections:179 - given scan urls are empty. set urls in the configuration
+logger.orgReflections.name = org.reflections.Reflections
+logger.orgReflections.level = off
+logger.sunReflections.name = sun.reflect.Reflection
+logger.sunReflections.level = off
+
+appender.console.type = Console
+appender.console.name = stdout
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
+appender.console.filter.threshold.type = ThresholdFilter
+appender.console.filter.threshold.level = error
+
+appender.rolling.type = RollingFile
+appender.rolling.name = FILE
+appender.rolling.layout.type = PatternLayout
+appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
+appender.rolling.filePattern = ./${filename}.%i
+appender.rolling.policy.type = SizeBasedTriggeringPolicy
+appender.rolling.policy.size = 4MB
+# Set the immediate flush to true (default)
+# appender.rolling.immediateFlush = true
+# Set the append to true (default), should not overwrite
+# appender.rolling.append=true

WindowsInstaller/Install Files/ntpcfg.bat

@@ -0,0 +1,33 @@
+@echo off
+
+:: BatchGotAdmin
+:-------------------------------------
+REM  --> Check for permissions
+>nul 2>&1 "%SYSTEMROOT%\system32\cacls.exe" "%SYSTEMROOT%\system32\config\system"
+
+REM --> If error flag set, we do not have admin.
+if '%errorlevel%' NEQ '0' (
+    echo Requesting administrative privileges...
+    goto UACPrompt
+) else ( goto gotAdmin )
+
+:UACPrompt
+    echo Set UAC = CreateObject^("Shell.Application"^) > "%temp%\getadmin.vbs"
+    echo UAC.ShellExecute "%~s0", "", "", "runas", 1 >> "%temp%\getadmin.vbs"
+
+    "%temp%\getadmin.vbs"
+    exit /B
+
+:gotAdmin
+    if exist "%temp%\getadmin.vbs" ( del "%temp%\getadmin.vbs" )
+    pushd "%CD%"
+    CD /D "%~dp0"
+:--------------------------------------
+
+net stop "Windows Time"
+
+w32tm /config "/manualpeerlist:pool.ntp.org 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org 3.pool.ntp.org cn.pool.ntp.org 0.cn.pool.ntp.org 1.cn.pool.ntp.org 2.cn.pool.ntp.org 3.cn.pool.ntp.org"
+
+net start "Windows Time"
+
+sc config w32time start= auto