Arweave-json-uploader

A simple node.js script to upload json data to the Arweave Blockchain via CSV. The script comes as is and the author is not liable for lost AR tokens due to misuse.

Prerequisites

0. ts-node
1. yarn
2. nvm
3. You will need an Arweave wallet with funds to actually write to the blockchain. You can create a wallet with a small amount of AR tokens following this guide. You can also create an ArConnect wallet as an alternative but you will not get any free AR tokens with that route.

Recommended Prerequisite

The arweave-json-uploader supports running against a locally running testweave for testing purposes. It is highly recommended you have run your upload against the testweave before committing to writing against the Arweave mainnet as writes are permanent and cost actual AR token. Follow the instructions in the official ArweaveTeam teastweave-docker repo to get yourself up and running with a local testweave docker container. Instructions on how to install docker on your system is provided in the guide above.

Note: You do not have to worry about the TestWeave SDK as the uploader script already has it integrated.

Usage

Setting your environment variables

The script assumes you have the following environment variables defined in your system: ARWEAVE_ADDRESS and ARWEAVE_KEY. To set these variables in your system (assuming a MacOS or Linux OS), do the following:

0. Find the Arweave wallet json file you set up in step 3 of the prerequisites, it will look like arweave-key-yourPublicKey.json. Copy your public key from the file name (i.e. the yourPublicKey portion)
1. Open up a terminal and go to your root directory via cd ~
2. In your terminal you will want to open your .zshrc or .bashrc file (depending on what terminal you use) via vim .zshrc or vim .bashrc respectively. If you don't have any of those file(s) defined already the above command(s) will handle creating them for you.
3. Type i to enter edit mode and add the following to your file: export ARWEAVE_ADDRESS='put your public key here'
4. Open your arweave-key-yourPublicKey.json file with an editor (ATOM, Sublime, VSCode, etc...) and copy the entire contents.
5. Go back to your terminal with VIM open and add the following entry underneath your ARWEAVE_ADDRESS: export ARWEAVE_KEY='the json you just copied'
6. Press the esc key on your keyboard to exit editing mode, type :x and press enter to save the file and close VIM at the same time.
7. Close your terminal window and open another one so that your changes can be reflected. Type echo $ARWEAVE_ADDRESS && echo $ARWEAVE_KEY and press enter. If you did things correctly you should see you Arweave public address and key printed on screen.

Note: With the above complete you will have stored you public and private keys locally on your machine. As always never disclose your private key to anyone so that you can keep your funds safe.

For more information about setting up environment variables you can check out this blog post

How to build

0. Run nvm install in the project root directory
1. Run yarn to pull in the dependencies

Content source

The script uses the resources/sample.csv as the source of content when preforming an upload. A given row represents the data (the actual contents of the JSON) portion of what will be uploaded to the weave. Each row also represents the metadata that wil be attached to the data via tags(key/value pairs) where the headers of the CSV represent the keys (Please see the developer api docs for more information). Please adjust this file as needed with the data you wish to upload; however, this script is only intended to upload JSON in it's current form so you have to leave the Content-Type header and application/json value in each row so that things work as expected. There may be future expansions to the script to allow more content types.

You might be asking why the same data is being uploaded as the main content and as metadata at the same time. The author took this approach due to the fact that the Arweave graphQL api does not support querying for data specifically and wanted a way for data to be searchable/available through a single entry point.

Important: As a caveat to above design decision one must note that Arweave has a max limit of 2048 bytes total for all tags in a given upload (so in our case a given row in our CSV). If this doesn't work for your needs, please feel free to fork this repo and mold the script to your liking.

Command Line Options

The script runs entirely through the command line and can be run with the following command and option below:

Options

• -d, --dryrun <boolean> : Run the script in dryrun mode which will not upload to the blockchain giving you a chance to observe requests before committing to uploading. Defaults to true if not passed
• -t, --testweave <boolean> : Run uploads against a locally running testweave. Defaults to false if not passed

How to run the script

0. Open up a terminal and traverse to the project root i.e. cd some/path/to/arweave-uploader
1. Replace the content of resources/sample.csv with the data of your liking, making sure to keep Content-Type in the header and application/json as the first value in each row.
2. Run ts-node uploader.ts -d true to run the script in dryrun mode to observe how your requests will look like without actually uploading.
3. Observe the logs/service.log file to see if your data looks like you expect it to
4. (Optional but recommend) Open a separate terminal and traverse to where the testweave-docker repo is located i.e. cd some/path/to/testweave-docker.
5. (Optional but recommend) In the testweave-docker project root run docker-compose up to start up the testweave node.
6. (Optional but recommend) Go back to the original terminal window you opened with the arweave-uploader project root and run ts-node uploader.ts -d false -t true to begin an actual upload to your local testweave. If you notice your testweave docker container hanging on operations i've found that deleting the container and starting it again with docker-compose up resolves the issue.
7. (Optional but recommend) Once the upload is finished, observe the logs/service.log file to see the results
8. Assuming you are satisfied with your dryrun and/or testweave uploads you can now upload to the Arweave mainnet via ts-node uploader.ts -d false -t false (Warning: This step will actually consume AR tokens from your wallet)
9. Once the upload is finished, observe the logs/service.log file to see the results

Checking upload results

After running an upload, the logs in resources/service.log will provide you information around the actual transaction id's of each upload, the status of the uploads, as well as a link to either view you transaction on viewblock (if you uploaded to the mainnet) or the actual data on your local running testweave (if you uploaded to your testweave docker instance). Please note that when uploading to the mainnet, as with any other blockchain, it takes time for participants to confirm/verify your transaction so you might not see your data show up right away. Be patient and check back intermittently until the data finally shows up on chain. You can also check your wallet to see if the transaction costs have been reflected on your account.

In general you can view the actual contents of your upload by hitting https://arweave.net/{transactionId} in your browser. You can also use the graphql endpoint to query against the weave. A example relevant to this project is searching for a given transaction using tags as a filter

query {
    transactions(
        tags: [
          {
            name: "key1",
            values: ["value1"]
          },
          {
            name: "key2",
            values: ["value2"]
        }
        ]
    ) {
        edges {
            node {
                id
              tags {
                name
                value
              }
            }
        }
    }
}

Please view the graphQL documentation or the http api docs for more examples on how/what your can query.

See Issues or have a feature in mind?

If you see any bugs during use or have a cool feature request, please don't hesistate to create an issue :)

Found the script helpful?

If you found the script helpful to you and would like me to continue building/support the project consider a small (totally optional) ETH donation to the following Ethereum address: 0x35Bd29AF1DeD41Aacd561bc1E4eEf1E1EA852D77

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Narb 🚇 💻

This project follows the all-contributors specification. Contributions of any kind welcome!