nexpo-backend-nova

A rebuild of the nexpo backend with C#, currently in use for the ARKAD career fair as of 2023.

This is the backend that supports the Nexpo app and in the future hopefully the new nexpo website.

Introduction

Project Navigation:

Google Drive: Stores meeting protocols and image source files of the figures used in the wiki. (Requires a tlth google account to access)

Trello Board: Provides an overview of all issues, their progress and who's responsible. (Requires access)

Getting started

1. Download the required development tools
   1. Download Docker
   2. Download the .NET SDK
2. Start the backend, using the method that fits you best

• see Setup development environment for options and tutorials

3. Start developing! See System Overview for additional help

Table of Contents

• Introduction
  • Project Navigation:
  • Getting started
  • Table of Contents

---

• Setup development environment
  • runBackend.sh
    • Common causes of errors with ./runBackend.sh
  • Stand Alone
  • Docker & Visual Studio

---

• System Overview
  • Overall Architecture
  • Relations between Model tables
    • ER-Diagram
    • Relations

---

• Testing and developing tools
  • Setup Test Environment

---

• Setting up production server
  • View endpoints in swagger
  • Install docker on amazon linux 2 (a lot of issues)

---

• Update Database models
  • Changing or adding a Model
  • Create backup of the db
  • Restore db from backup
  • Generate migration-script
  • Apply migration to db

---

**Small footnote to whoever is updating this table of contents:

• get "All in one markdown -> ctrl + shift + p -> update table of contents**

Setup development environment

The main configuration methods for setting up the development environment

• Choose a method that suits you best:

1. runBackend.sh
   • (Recommended for Unix systems - also works for WSL)
2. Docker & Visual Studio
   • (Recommended for Unix systems)
3. Stand Alone
   • (Recommended for Unix systems, if the the bash script does not work - for example due to issues with WSL)
   • (no Docker, and you can choose any IDE you like, for example Visual Studio Code)

runBackend.sh

• Recommended for Unix systems

1. Install Docker

2. Install .NET SDK

   1. It is available here

3. Give the script execution permissions

   1. Run the following command to give the script execution permissions:

chmod +x runBackend.sh

1. Run the script:

sudo ./runBackend.sh

Common causes of errors with ./runBackend.sh

(Note, much of these solutions are also relevant to the standalone solution)

• You are not using sudo in front of sudo ./runBackend.sh
  • Alternativly you are not using sudo in front of docker commands
• Controll center uses port 5000 on MacOS
  • Verify this with lsof -iTCP -sTCP:LISTEN -n -P
  • To take controll over the port, run pkill ControlCenter; nc -l 5000, and kill the command after a few seconds
• Dotnet or Docker is not installed (correctly)
  • To test this, try if the following commands give a version

docker -v
dotnet --version

• The configured dotnet version of the backend is not the same as your downloaded dotnet version

  • A quick way to temporarily fix this is to simply change what version the backend uses:
  1. Run the following command and note your dotnet version:

  dotnet --version
  

  2. Change the TargetFramework in Nexpo/Nexpo.csproj to match your dotnet version
     • For example if you have dotnet version 7.0.105, change to dotnet 7 in Nexpo/Nexpo.csproj, meaning replace with the following line:

  <TargetFramework>net7.0</TargetFramework>
  

  3. WSL1 causes some complications with docker. Use the following command to test your WSL version, and update if you still can not get Docker to work

  wsl -v
  

• If none if the above fixes the problem try making an standalone launch using the following command: sudo ./runBackend -s

Help regarding runBackend.sh

The command ./runBackend -h is useful for receiving help regarding the bash script.

Stand Alone

1. Install Docker Desktop
   1. You can get it for Windows and Mac here
2. Containerize the database, by running:

sudo docker run -d --name nexpo_database -p 5432:5432 -e POSTGRES_USER=nexpo -e POSTGRES_PASSWORD=nexpo postgres:14

Click to Expand for additional information

1. Note: On windows, simply skip using sudo
2. The default development database connection tries to access the database nexpo on localhost:5432 using the credentials nexpo:nexpo. The nexpo user of course needs the correct permissions on the database and if you change anything in the setup make sure to update the connection string as well.
3. It will pull down the correct PostgreSQL server and set it up as we want it. Keep in mind though that no persistent volume is added to the container so don't do this in production.

3. Install .NET SDK
   1. It is available here
4. Run the backend
   1. Run the following command to run:

dotnet run --project Nexpo

Click to Expand for additional information

To avoid having to specify the project every time, you can also:
1. Change directories to the `Nexpo` project folder.
2. run `dotnet run` (without the `--project Nexpo` flag)

How to visualize DBeaver

1. Run previous command for running database in docker without Docker Desktop.
2. Download DBeaver https://dbeaver.io/download/
3. Connect to open postgres connection with settings Postgresql and default 5432 default port. db_name, user, password as nexpo

Docker & Visual Studio

• Recommended for Windows

1. Install Docker Desktop

   1. You can get it for Windows and Mac here

2. Install Visual Studio

   1. Use the Community Edition if you don't have a license. During installation, select the "ASP.NET and web development" toolbox when given the choice to enable good support for Docker and the application.

3. Open the solution (the code) in Visual Studio

   1. Open Visual Studio and open the solution file Nexpo.sln.

4. Make sure the docker-compose "project" is selected as the startup project

   1. If not already selected as the startup project, right click docker-compose in the Solution Explorer and select "Set as Startup Project".

5. Run the backend

   1. A database with automatically be created

---

System Overview

Central Architecture

The overall architecture can currently be split into 6 components with different responsibilities. They are as follows:

1. Controller: \ Receives and responds to http-requests by calling on appropriate methods in the other components to generate the desired outcome. To control the format of the input and output, may requests and db responses be converted to DTO:s before being forwarded to repositories or sent as a response.

2. Repository: \ Responsible for translating requests into queries against the model and converting query results to relevant data objects before returning them.

3. Model: \ C# representation of the the database tables.

4. DTO: \ Data Transfer Object that converts data to an object consisting of only relevant data. Can be used to prevent data leakage in http-responses or as an simplified method of moving data between different components.

5. Services: \ Responsible for functionality outside the manipulation and gathering of data in the database. This entails token & file management, password validation and email services.

6. Helpers: \ Consists of helper functions for the controller. Currently only converts claims to intelligible data.

Other resources

1. UploadToDB: Responsible for download and uploading files to the database.

• More documentation to come

2. Scripts:

• The project has Bash and Bat scripts for running the backend. The scripts are located in the root folder of the project.

• There are also Bash and Bat scripts for running the tests. (More information regarding is given when running the corresponding help command, eg: ./runTest.sh -help These scripts have the ability to:

  1. Start the database in a docker container

     • This also creates or updates the mock data in the database

  2. Run all tests

     • Note that the tests will fail during the first run of the tests, since the container has likely not had time to start up completly. Simply run the tests again.

  3. Run some of the tests, by specifing the class or controller to test. 3. Tests

• The tests are located in the Nexpo.Tests project. They currently only test the controller and services, and assumes that everything else (which is dependent) works as intended.

• There is a TestUtils available, which is helper class for logging in to the system while testing. These are dependent on the mockdata in ApplicationDBContext

There are currently more than 200 tests, which might be demanding for the computer. Therefore the ./runTest.sh -help decalres ways to minimize this issue. Namely, by only running a fraction of the tests:

Options:
  -start            Reset the database before running tests
                    This needs to be done when the database seeding has changed
                    The first time this is done, the tests will fail
                    Ergo, run needs to be run twice

  Run twice after -start:
  -run Run all tests
  -run <class_name> Run the specified test class
  -run <controller_name> Run the test class for the specified controller
  -run <name>       Run the test class for the specified controller

More in-depth resources:

Architecting-Modern-Web-Applications-with-ASP.NET-Core-and-Azure.pdf

NET-Microservices-Architecture-for-Containerized-NET-Applications.pdf

Relations between Model tables

ER-Diagram

Relations

Current relations between the tables in the model:

User(Id, Email, PasswordHash, Role, FirstName, LastName, PhoneNr, FoodPreferences, hasProfilePicture, hasCV, ProfilePictureUrl, CompanyId)

Student(Id, Programme, ResumeEnUrl, ResumeSvUrl, LinkedIn, MasterTitle, Year, UserId)

Company(Id, Name, Description, DidYouKnow, LogoUrl, Website, HostName, HostEmail, HostPhone, DesiredDegrees, DesiredProgramme, Positions, Industries)

StudentSessionTimeslot(Id, Start, End, Location, StudentId, CompanyId)

StudentSessionApplication(Id, Motivation, Status, Booked, StudentId, CompanyId)

Event(Id, Name, Description, Date, Start, End, Location, Host, Language, Capacity, TicketCount)

Ticket(Id, Code, PhotoOk, isConsumed, EventId, UserId)

---

sudo docker run -d --name nexpo_database -p 5432:5432 -e POSTGRES_USER=nexpo -e POSTGRES_PASSWORD=nexpo postgres:14

sudo dotnet test Nexpo.Tests/

Testing and developing tools

Setup Test Environment

No matter the chosen setup method, it´s required to start an external database server before running the tests for them to pass. This is due to some tests utilizing black-box testing through testing against the controllers. It may take a while for the container to populate the tables with the example data, so if most controller-tests fail during the first run try to run them again.

NOTE: Running docker-compose in Visual Studio and then the tests do for some reason not work. To run the tests:

sudo docker run -d --name nexpo_database -p 5432:5432 -e POSTGRES_USER=nexpo -e POSTGRES_PASSWORD=nexpo postgres:14

sudo dotnet test Nexpo.Tests/

Setting up production server

View endpoints in swagger

Swagger allows you to see the specifications of an API, including the endpoints, request parameters, response formats, and authentication methods. It provides an user interface for interacting with the API and testing its functionality. With Swagger, you can visualize and document your API in a standardized way, making it easier for developers to understand and use your API.

To open the application in swagger:

1. Start the backend
2. Go to http://localhost:5000/swagger/index.html

Install docker on amazon linux 2 (a lot of issues)

port 80 being used... service nginx stop

Update Database models

The classes stored in the Models directory are the skeleton for the database. We use something called Code First to define our database structure in code relationships and then generate the database modifications automatically.

Changing or adding a Model

If you need to change a Model (adding a new field for example) or create a new one it is as simple as creating a new class in the Models directory. If you are adding a new Model you need to add the supporting classes (the repositories) as well and add the correct bindings that make them a new table. Take a look at how the others are connected, it should only be about two lines of code in ApplicationDbContext to add it as a table and then in Startup to add the repository to the dependency injector.

After the code changes you have to generate the database update code, the so called Migration. A migration is a small script that transforms the database from the old state to the new to match what you added or removed in the code. For example, if you added a field to a Model, the transformation will generate the necessary SQL to alter the table to include the new column. Migrations can also be reverted since they contain the information to undo the changes. To create a new migration for your changes, run the following command:

dotnet ef migrations add <a descriptive name for your changes>

After the migration is created you can take a look at the generated code to see if it is correct or if you need to make some changes (to the Model, don't edit the generated migration). The migration is applied on development server startup but if you want to apply it manually you can run:

dotnet ef database update

Create backup of the db

To create a backup dump of the Postgres database, run the following shell command:

docker exec -t name_of_db_container pg_dumpall -c -U nexpo > name_of_dump.sql

NOTE: Store the dump file on a secure and private instance as it contains sensitive data about the content of the db.

Restore db from backup

To restore the db from a backup dump, run the following shell command:

cat name_of_dump.sql | docker exec -i name_of_db_container psql -U nexpo

Generate migration-script

To generate a script that applies generated migrations to the db, run the following shell command:

dotnet ef migrations script --idempotent > name_of_migration.sql

NOTE: Remove the two first lines in the generated script as they are simply just output from the build-process.

Apply migration to db

To apply the migration to the production db, use the generated script and run the following shell command:

cat name_of_migration.sql | docker exec -i name_of_db_container psql -U nexpo

Deploying and updating the Backend

The backend is currently hosted on AWS EC2 as three docker containers. Additionally, profile pictures and CVs are stored in AWS S3.

Updating the Database

There are several ways that the database can be updated.

Uploading via EC2

The most straightforward one being simply doing it in the EC2 instance. Since the database is hosted in EC2, PostgreSQLcommands can be used to update the database. If you have found the database… good. If not, go to EC2 and enter

docker exec -it <CONTAINER ID OF postgres:14> bash

psql -W nexpo -U nexpo

• Enter the password

Uploading via python scripts

• In the folder ‘UploadToDB’ with a collection of python scripts and parsers that can be used to automate meticulous tasks. Note that some of them do not currently work, or work badly, as of IT22. (sorry)

Updating the Backend

Updating the backend endpoints

The backend currently running is available in Github at /home/ec2-user/nexpo/nexpo-backend-nova It is updated by pulling the updated Github repo. The keys are (obviously) not in the Github repo, but should automatically be filled in. Make sure that this is the case. Also remember to migrate:

dotnet ef migrations script
dotnet ef database update

Then update the containers:

docker-compose -p nexpo build
docker-compose -p nexpo up -d

• This should create three docker containers docker ps