Log&to-dos

# 100 Days Of Code - Log

### Day 1: 12, May 2024 

**Today's Progress**: Started adding web3 to a node/react project. Begun by adding Privy login to the embed wallet of the submissions app

**Thoughts:** This was painfully easy... but may not work long term. Currently my app is on create-react-app which requires Typescript 5<. Privy requires typescript 5>. I tried to set some overrides for a while but then gave up.

**Link to work:** pushed 

### Day2
**Today's Progress**:: I noticed yesterday that the submission period check (which was meant to be open all week), wasn't working on Sundays yesterday. I fixed it by working out that JS starts Sundays at zero (to saturday's 6) instead of Monday's at zero. Once adjusted, it worked. I then moved back to privy and specificially the 'authenticated' option. I added an IF statement
**Thoughts:** - Privy is great. 

### Day3
**Today's Progress**:: mostly back and forth with gpt on the best way to run the voting component of submissions; onchain or offchain. The hypothesis for now is a hybrid approach. Vote power is calculated via a smart contract based on tokens in a wallet. Vote tallying & result itself is via our server due to our requirement to have a centralised veto for now. We can transfer this to onchain vote plus multisig veto later... but lets not deal with gas and robustness problems right now.After this decision, I looked for a very basic vote power snapshot contract on base sepolia but couldn't find one. So for now I'll write my own. I then spent the rest of my time, setting up my environment for contract work. That took ages. Just in 5 mins I took a template of the votepower.sol contract and deploy.js gpt gave me and shoved it in. I'll pick up reviewing these tomorrow
**Thoughts:** - Dev environment versioning sucks... but gpt does get there in the end. I'm not entirely happy using my own voting snapshot contract. this must exist already on base sepolia so will keep searching

### Day4
**Today's Progress**:: went back through the templates of votepower.sol / deploy.js. Commented up to make sure I understand it fully. I then tried to test the contracts via command line/hardhat... kept getting an error ('TypeError: votePower.deployed is not a function'). No solution yet. UPDATE - founda  solution. It was due to ethers6 updating how we call stuff (e.g not contract factory). updates made to deploy.js to adjust. Was able too deploy and then check vote power on an address (yay)
**Thoughts:** - hardhat is tricky... but helpful. 

### Day5
**Today's Progress**:: swapped out degen - which I had zero vote power - for a sepolia erc-20 token that I deployed. Added code to my server.ts to grab the users vote power . Server runs but fails in the front-end. Hard to test with postman too as need a wallet. Will implement front-end code before testing. 
**Thoughts:** - 

### Day6
**Today's Progress**:: should have stayed in the backend. Tonne of errors. Givev up until tomorrow

### Day7
**Today's Progress**:: fixed the backend issues. It was a mixture of there being no ABI input, the alchemykey not being called from dotenv correctly, the getVotePower function not storing data as the right combinations of string/number/big number etc. Once I did these changes it worked. however, I realised the vote power can be used continually to vote. I then added backend functionality to only let a wallet vote once in a set period. This will do for now (though note that when the token is transferable, this won't work as they can just move the vote power to a different address). The front-end now also got updated to send a nice notification if they try and vote twice (before the app just crashed)
**Thoughts:** - lesson here is not just to try and do all the backend, frontend code all at once and then debug. A lot easier to just do it piece by piece and track errors as I go. 

### Day8
**Today's Progress**: set up routers so we can move to a multipage site. Updated the homepage.tsx to work with our backend (i.e. display a 'no market live yet message', then receive a title for a proxy, voted on prediction market from the nextweek page, display that title once its chosen). Now will spend the rest of the day deciding how to do the prediction market aspect (onchain, offchain, hybrid). 
**Thoughts:** - this isn't going to be easy

### Day9
**Today's Progress**: spent the day in research with gpt instead of implementation. In short, no point me going further with the front-end / databasing until I know how i'm doing the hardest part. The prediction market itself. It seems Omen (gnosis) does onchain AMM market, offchain matching, onchain trading of shares as 1155s, onchain execution. It seems polymarket, does offchain market, offchain matching and trading of shares onchain. I got some templates generated for how to do the making with smart contracts, the matching with a server and the trading onchain. I'll ask for some advice today and then implement tomorrow.
**Thoughts:** - This is getting very deep. Might try and get some advice before I pick a path
### Day10
**Today's Progress**: leaving the homepage prediction market for now. Gonna get some advice on it first. For now, I'm going to setup the database so the data for submissions at least is persistent. Decided on using mongodb mostly as it uses json which is easier for me to pick up than postgresql. I imported mongoose. set my dotenv variables and then connected successfully to mongo in my server. I also then pasted some templates for making my get / post submissions working in mongo. They worked for storing and posting from postman... BUT the votign and setting live market now failed. Something to do with the ID changing (the id changed location in the api's deliverable. This probably just needs some tweaks to the vote/reset functions)
**Thoughts:** - databasing seemed easy for submissions. Again most worried for when this scales up to the prediction market/order matching. Onchain might be better
### Day11
**Today's Progress**: okay, I updated the vote, set-live-market and get-winner functionality in server.ts. They now work with mongo and persistently. I noticed that the User can vote twice in a submission period again if the server resets.. so created a mongodb model and stored the check on user votes in the database too.
to-do gaps in functionality. 1. Reset will reset submissions forever. 2. Front-end is compatible with voting yet

### Day12
**Today's Progress**: didn't do much today as was travelling for work. Spent the plane ride architecting the AMM. Decided I'm going to implement it myself once done with with the database integration

### Day13
**Today's Progress**: okay, back to databasing. Targeted making front-end compatible so voting works. Took a while but saw that the submissions object now had 'submissions._id' instead of submissions.id which meant the id was undefined. Just changing the html though didn't work. Our server.ts interface needed updating too. This now works

### Day14
**Today's Progress**: okay, took the plunge and decided to start implementing contracts for the AMM. I've just taken some templates for .sol files. Will review them tomorrow

### Day15
**Today's Progress**: travel day

### Day16
**Today's Progress**: forgot to push this, but yesterday I reviewed how the proposed contracts are setup and then adjusted the deploy.js. I haven't tried compiling yet. Will bug fix tomorrow

### Day17
**Today's Progress**: 'had to wrap conditional tokens in a wrapper with a higher contract. Tweaked other compatibility stuff. Contracts now compiled and deployed. Front-end integration next'


### Day18
**Today's Progress**: updated server.ts to push my live market onchain on submission period end. I still get an error when I do this though so need to check this next'

### Day19
**Today's Progress**:'still trying to sort out the server calls of the contracts. Function calls are named wrong. Still erroring but am closer'

### Day20
**Today's Progress**:'good news. The condition for the conditional token is now preparing. The problem was we passed the ERC20 address to the tokenswrapper. We needed to pass a ConditionalTokens.sol into the wrapper. I created this, referenced IConditionalTokens, and got it working. Now The MarketMaker is erroring out. So will move onto that'
**Extra Progress**- so turns out there was still error. It worked on the first server run but failed everytime after as questionID has to be unique each time. I made the ID unique each time by combining the hash with a timestamp

### Day21
**Today's Progress**:able to set a live market in marketmaker contract. Solution to the bug was converting the livemarket.endTime to a unix timestamp to stop bignumber issues. Not moving onto front-end yet. First i want to write some scripts to make sure the contracts work once deployed by our server. 
**Extra Progress**- I was dumb. Forgot to actually add the buying/trading of shares via an AMM in marketmaker.sol. Lots to do here so going to push where I am so far
**Extra Progress2.0**- got a new marketmaker.sol to add buying/selling. An overrand fee setup and initial liquidity. The script runs to do this. Next steps, still need to extend to remove liquidity. Also need to update server.ts to account. 
**Extra Progress3.0**- busy day. Added remove liqudity in the marketmaker.sol. then updated server.ts to work with our new marketmakers. Not 100% I've covered everything that makes a prediction market work... but I think I'll only find more flaws in it by progressing for now. I'm going to move onto front-end tomorrow

### Day22
**Today's Progress**: front-end functionality works to add/revoke liquidity and buy shares. However the front-end doesnt pull the latest live market if the server stops. Need to tackle that next. 

### Day23
**Today's Progress**: made the market persistent between server restarts by storing the livemarket variable in our database. Next steps, I want to show more data in the front-end about the markets. Maybe initially how much liquidity is in the market. Second, how many shares have been bought. This will likely mean I need to return to the contract. Right now the liquidity pool and shares are seperate. There should probably be a link. Finally, Id like the front-end to be able to display old markets. 

### Day24
**Today's Progress**: mostly research today. As mentioned, right now we buy/sell outcome shares but they dont create a price nor a probability. The different methods for doing this are CPMM/LS-LMSR. Im going to go with CPMM as its simple and encourages liqudity. Ive taken gpts template for adding this functionality and pasted it in. Ill review and tweak it to my needs tomorrow
### Day25
**Today's Progress**: getting a panic overflow error. Annoying. Going to pick it up when back from a long weekend. 

### Day26
**Today's Progress**: none

### Day27
**Today's Progress**: none

### Day28
**Today's Progress**: none

### Day29
**Today's Progress**: none

### Day30
**Today's Progress**: back online after a weekend away. Still getting the error. After some time out, Im thinking of redoing my contract setup. Thisd mean a lot of redone work... but conditionaltoken wrapping etc is bloating my contracts. Too many bugs and errors. I need somethiing simpler. e.g. looking at copying this but adding liquidity - https://github.com/UMAprotocol/dev-quickstart-oov3/blob/master/src/PredictionMarket.sol

### Day31
**Today's Progress**: Picked a new contract. but having massive version issues. My start from scratch in a new environment tomorrow to see if that helps

### Day32
**Today's Progress**: I split the contract into two. the market functionality with manual resolve. And another contract to be turned on later for the oracle. This seems to have mitigated the contracts required and whichever version was screwing things up. Its compiling now so next step is to test deploying it/functions with scripts

### Day33
**Today's Progress**: deploy.js failed due to contract size. I had to move some variables to bytes32 (could bite me later) and not compile the oracle to get it under the limit. It now deploys. Next up is test-market.js

### Day34
**Today's Progress**: test-market.js showed me that changing to bytes32 would be horrendous. I changed it back. Hit some other snags on test-market.js but worked through them. Now contract deploys and runs. Before I move on, next step I want to review the required bond / reward. I mean, if Im not using an oracle yet, this isnt necessary

### Day35
**Today's Progress**: day off

### Day36
**Today's Progress**: Okay so a good day. Hit contract size issues again. Managed to cut the contract size sdown by half by moving createToken function to a new MarketManager.sol. I then added a new contract for trading and liquidity so our server can come up with the outcometoken price. These contracts all deploy now. I still need to test them next

### Day37
**Today's Progress**:hit another wall. Realised liquidityandtrading wasnt correct and replicated a lot of bad funcitonality. Made me question my approach overall again. Will pick up tomorrow

### Day38+ 39
**Today's Progress**:- so Ive gone back to gnosis contracts. I also think I need to spend some time doing the tutorial with them pre integrating here. Going to create a new repo in 100-days-of-code to do this tutorial. Will leave daily notes there for now and come back here when done - https://docs.gnosis.io/conditionaltokens/docs/pmtutorial3'
[main 799084f] so Ive gone back to gnosis contr

### Day40 + 41 :
**Today's Progress**: I ended up staying in this repo and redeploying the gnosis contracts here. Took ages and tonnes of tweaks to make the contracts work with ethersv8 and new OZ. It compiles at least now and added a solidity optimiser to fix size. Tomorrow will pick up deploy.js / test-market.js. Im sure there will be more bugs. Im changing more of these gnosis contracts than I would like so will pay a dev to review the changes at some point
### Day42:
**Today's Progress**: I had to reformat some of the contracts. GPT led me down a path of deploying the predictionmarket myself but actually that is what the factory is for (duh). Got that compiling and then finally deploying. Now to make sure all the functionality is there in deployment
### Day43:
**Today's Progress**: got the factory to deploy the market maker. However stuck on a panic overflow in the addFunding function. Will come back to it
### Day44:
**Today's Progress**: Im getting completely destroyed by the panic overflow error. Cant get any events or errors to come from the contract. My gut is the contract isnt being setup correctly by the factory. I think Im going to have to go back to removing the factory and initialising / constructing it myself... grr
### Day45:
**Today's Progress**: Finally got the testmarket script runing. The panic overflow was due to a for loop using i instead of int() in splitPositionThroughAllConditions. Then got major stuck on the sell function. The outcometokens balance required to sell an amount was way higher than thought. A for loop eventually solved it but I still need to understand it better and how maxSellOutcomeTokens works. Either way, it runs end to end. Next step is to check my app functionality against these calls and make sure all functionality is included
### Day46:
**Today's Progress**: I spent today really understanding the solution I found yesterady, cleaning it up and adding final functionalities to our script. First I added console.logs to understand every token movement when buy, addFunding and their opposites are run. I then added a script to work out the price of the outcome tokens (i.e. their probabililties). To do this I had to make getPoolbalances public in the contract which I didnt like. I still wonder how Gnosis gets prices out. Finally then I added a fee to the script to see what would happen. This also meant using any last untouched functions in the contract to check totalfees, fees I can claim, and withdraw fees. These all ran nicely, however the contract after the withdrawFee function did hit nonce and gas issues. Annoying but Ill come back to it
### Day47:
**Today's Progress**: Added some starter code to refactor my server.ts in order to work with my newly deployed base gnosis contracts. Lots of iterations and testing to go, but a good start
### Day48:
**Today's Progress**: none
### Day49:
**Today's Progress**: Great, so took some debugging my server can now set a market, add liquidity, remove liquidity, buy outcome tokens, sell outcomes. I still need to add 1 - the rest of the functionality from testmarket.js (i.e. collectprices, withdrawFees, redeempositions, resolve the condition) and then 2 - add new stuff like checking how much liquidity is in the market, how many buy or sell shares. etc
### Day50:
**Today's Progress**: Cool so I added redeem positions and resolve the condition to my endpoints. Took some back and forth cos it turns out I wasnt wiping the last prediction market correctly. Solved now. Next I should have the easier read functions of collectPrices, and check liquidity in the market. Withdraw fees didnt work in test-market.js so thats final boss pre front end. 
### Day51:
**Today's Progress**:added check liquidity in the market and basic pricing. 

### Day52:
**Today's Progress**: Added nonce manager already. We then allowed the server error handling so it fails more gracefully across endpoints. e.g. liquidity to be zero on read so it errored more cleanly. allow endpoints to run when a market is closed. track state fo market resolve/redemption so buy,sell and liquidity only runs when contracts allow. error correctly when you try and buy outcome tokens without liquidity or sell tokens without buying. similar with buying/ removal without liquiduity. Finally, also that you can't resolve twice to stop an ugly error. 

### Day53:
**Today's Progress**: confirmed the gnosis contracts handle the user positions correctly. Still unsure why we can call it so many times. Will pick it up tomorrow 

### Day54:
**Today's Progress**: confirmed why gnosis allows people to redeem multiple times. Its to allow liquidity providers to redeem this at any point. I then added two endpoints for these providers to read and exit with any fees gained. Finaly I rebooted up my front-end. The submissions page still sets the market in our backend which is good. 

User held outcometokens from buying outcome 1
user redeemed these outcome tokens for 2 erc-20 collateral
contracts allow user to then remove liquidity (even after redemption) so they received more outcome tokens 
user then must be allowed to redeem multiple times to get back the erc-20 for these last outcome tokens for 2 erc-20

### Day55:
**Today's Progress**: so gnosis sell function for some reason requires the user to know how much liquidity they want back (instead of how many outcome tokens they want to sell). I tried implementing a method where wed do that mental effort for them. That didnt work at all. Ill keep going for now how they want it... but I dont like how it works currently.

### Day56:
**Today's Progress**: moved to the market front end. 3 out of 4 endpoints worked. Buy, add and remove liquidity. Basic market details also appears. Sell ofc didnt and Im having issues even from my server. I might have broken somehting yesterday. Will pick up tomorrow

### Day57:
**Today's Progress**: updated state to store each input field individually so they aren't all updated onChange together. Also added a maxOutcomeTokenToSell input field for now. Still hate this UX for the user
### Day58:
**Today's Progress**: I've had massive problems with calc-sell aritmatic. when I removed safeMath to work with new versions of eth, something must have broken. My brain is going to explode with this

TO DO IDEAS:
Essentials for Later Development
Persistent Data Storage:
Database Integration: Replace the in-memory storage with a database like MongoDB, PostgreSQL, or similar. This will ensure that your data persists across server restarts and can be scaled as your application grows.
Data Models: Define robust data models/schema for submissions and votes. This helps in managing and querying data more efficiently.
Authentication and Authorization:
Allow vote power based on onchain tokens in embedded or EOA wallet
Robust Error Handling: Implement comprehensive error handling in the backend to manage and respond to errors gracefully.
Data Validation: Validate incoming data to prevent incorrect or malicious data from being processed.
Nice-to-Haves for Enhanced Functionality
Real-Time Updates:
WebSockets or Polling: Implement WebSockets for a real-time user experience, especially for displaying live voting results and market data updates.
Push Notifications: Inform users about important events (e.g., market changes, voting results) through real-time notifications.
Advanced Market Features:
Dynamic Pricing Algorithm: Implement or integrate a more complex algorithm for better prediction market dynamics.
Historical Data Analysis: Provide historical analysis of market predictions and outcomes to enhance user trust and engagement.
User Experience Improvements:
Interactive UI Components: Enhance the frontend with interactive charts and visualizations using libraries like D3.js or Chart.js for market data.
Mobile Optimization: Ensure the application is fully responsive and optimized for mobile devices.
Security Enhancements:
Rate Limiting: Prevent abuse by limiting how frequently a user can make requests to the API.
Audit and Monitoring: Implement logging and monitoring to detect unusual activities or performance issues.
Scalability Considerations:
Load Balancing: Prepare for scaling your application with load balancing techniques.
Microservices Architecture: If expecting significant growth, consider adopting a microservices architecture to make scaling individual components of your app easier.