From 9017e9593572daf55180d448cb3a4a820e7efc18 Mon Sep 17 00:00:00 2001 From: Ben Garrett Date: Wed, 12 Jun 2024 17:59:29 +1000 Subject: [PATCH] simplified docs and removed redundant texts. --- docs/README.md | 11 ++- docs/database.md | 85 +++++++++++++++++ docs/patterns.md | 194 +++++++++++++++++++++++++++++++++++++++ docs/readme_db.md | 126 ------------------------- docs/readme_go.md | 108 ---------------------- docs/readme_handler.md | 180 ------------------------------------ docs/readme_model.md | 114 ----------------------- docs/todo.md | 54 +++-------- internal/tags/strings.go | 1 + 9 files changed, 303 insertions(+), 570 deletions(-) create mode 100644 docs/database.md create mode 100644 docs/patterns.md delete mode 100644 docs/readme_db.md delete mode 100644 docs/readme_go.md delete mode 100644 docs/readme_handler.md delete mode 100644 docs/readme_model.md diff --git a/docs/README.md b/docs/README.md index 086e7c78..0414eb7b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -237,12 +237,21 @@ task testr ### Documentation -The application configuration documentation can be generated and [viewed in a web browser](http://localhost:8090). +The application configuration documentation can be modified in [`doc.go`](../doc.go) and the changes regenerated and [previewed in a web browser](http://localhost:8090). ```sh task doc ``` +```go +// Copyright © 2023-2024 Ben Garrett. All rights reserved. + +/* + +The [Defacto2] application is a self-contained web server first devised in 2023. +It is built with the Go language and can be easily compiled for significant server +``` + ### Building the source code The source code can be built into a binary for the local machine. diff --git a/docs/database.md b/docs/database.md new file mode 100644 index 00000000..8b716be8 --- /dev/null +++ b/docs/database.md @@ -0,0 +1,85 @@ +# Database readme + +Previous iterations of the Defacto2 web application relied on MySQL for its database. But for this 2023, Go application rewrite, the site will use [PostgreSQL](https://www.postgresql.org). With the Defacto2 database schema, Postgre uses less memory and is far more performant with complex queries. + +Postgre is more strict about data types than MySQL. For example, inserting a string into a numeric column in Postgre will throw an error, whereas MySQL will convert the string to a number. Postgre has a more powerful query optimizer meaning queries often run faster with complex joins and subqueries. + +# Table and data changes to implement + +These are only suggestions and may not be necessary if they create too much work or complexity. + +- [X] Rename `defacto2-inno` database to `defacto2_ps` +- [ ] Rename `files` table to `release` or `releases` +- [ ] Create a `release_tests` table with a selection of 20 read-only records +- [ ] Rename `files.createdat`, `deleteat`, `updatedat` etc to `x_at` aka `create_at`... +- - [ ] __OR__ break convention and use `date_created`, `date_deleted`, `date_updated` etc. + +### [datatypes](https://www.postgresql.org/docs/current/datatype.html) differences + +- [ ] `CITEXT` type for case-insensitive character strings +- [ ] `files.filesize` should be converted to an `integer`, 4 bytes to permit a 2.147GB value +- [ ] `files.id` should be converted to a `serial` type +- [ ] There is no performance improvement for `fixed-length`, padded character types, etc, meaning strings can use `varchar`(n) or `text`. + +### Indexes + +- [ ] Create PostgreSQL *indexes* with case-sensitive strings for [optimal performance](https://wirekat.com/optimizing-sql-based-on-postgresql/)? +- [ ] Partial Indexes: Use partial indexes when you only need to index a subset of rows, such as, +- - `CREATE INDEX ON orders (order_date) WHERE status = 'SHIPPED'`; +- [ ] Over-Indexing: Creating too many indexes can slow down write operations, as each index needs to be updated on `INSERT`, `UPDATE`, or `DELETE` operations. +- [ ] Index Maintenance: Rebuild indexes periodically to deal with bloat using `REINDEX`. +- [ ] Indexing Join Columns: Index columns that are used in JOIN conditions to improve join performance. + > `combineGroup` and `(r Role) Distinct()` + + +### Future idea, _file archive content_ relationship table + +Create a relationship files table that contains the filename content within of every archive release. + +We could also include columns containing size in bytes, sha256 hash, text body for full text searching. + +This would replace the `file_zip_content` column and also, create a CLI tool to scan the archives to fill out this data. For saftey and code maintenance, the tool would need to be a separate program from the web server application. + +## Migration from MySQL to PostgreSQL + +This document describes how to migrate the Defacto2 MySQL database to PostgreSQL using [pgloader](https://pgloader.io/). Note, the migration is a one-time operation and should be run on a development or staging server before running on the production server. + +- `defacto2-inno` is the name of the MySQL database. +- `defacto2_ps` is the name of the PostgreSQL database, note the `_` in the name as opposed to `-`. + +Create a migration loader file named `migrate.load` with the following content, replacing the connection strings with your own database credentials: + +```sql +LOAD DATABASE + FROM mysql://root:example@localhost:3306/defacto2-inno?useSSL=false + INTO pgsql://root:example@localhost:5432/defacto2_ps + + WITH include drop, create tables + +ALTER SCHEMA 'source_db' RENAME TO 'public' +; +``` + +Run the migration using the following command: + +```sh +# run the migration +pgloader migrate.load + +# test the migration +postgres psql +``` + +```sql +# SELECT * FROM files; +``` + +A simple client application to interact with the migrated database is [Postbird](https://github.com/paxa/postbird). + +Some more resources: + +- [pgloader](https://pgloader.io/) +- [documentation](https://pgloader.readthedocs.io/en/latest/) +- [Docker hub](https://hub.docker.com/r/dimitri/pgloader/) +- [DigitalOcean how-to migrate](https://www.digitalocean.com/community/tutorials/how-to-migrate-mysql-database-to-postgres-using-pgloader) +- [Migrating from MySQL to PostgreSQL](https://pgloader.readthedocs.io/en/latest/tutorial/tutorial.html#migrating-from-mysql-to-postgresql) diff --git a/docs/patterns.md b/docs/patterns.md new file mode 100644 index 00000000..63f9d261 --- /dev/null +++ b/docs/patterns.md @@ -0,0 +1,194 @@ +# Patterns + +## New + +This is a collection of patterns and idioms that could be useful with the new releases of Go. + +- https://go.dev/blog/go1.22 +- https://tip.golang.org/doc/go1.22 +- https://go.dev/blog/go1.21 +- https://tip.golang.org/doc/go1.21 + +### Randomization in Go 1.22 + +New random package, [`math/rand/v2` package](https://pkg.go.dev/math/rand/v2). + +```go +// print a random letter +alpha := []string{ + "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", + "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"} +fmt.Println(alpha[rand.IntN(len(alpha))]) + +// print a random number between 0 and 100 +fmt.Println(rand.IntN(100)) +``` + +### New Null type in Go 1.22 + +New type `Null` for [database/sql package](https://pkg.go.dev/database/sql#Null). + +```go +var s Null[string] +err := db.QueryRow("SELECT name FROM foo WHERE id=?", id).Scan(&s) +... +if s.Valid { + // use s.V +} else { + // NULL value +} +``` + +### New version comparison in Go 1.22 + +New [`go/version` package](https://pkg.go.dev/go/version) for version comparison. + +```go +func Compare(x, y string) int +func IsValid(x string) bool +func Lang(x string) string +``` + +### New built-in functions: min, max, clear in Go 1.21 + +```go +x, y := 1, 100 +m := min(x, y) +M := max(x, y) +c := max(5.0, y, 10) + +alpha := []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m"} +clear(alpha) +``` + +### New slices package in Go 1.21 + +New slices package for slice manipulation which are more efficient than `sort.`. +https://pkg.go.dev/slices + +```go +smallInts := []int8{0, 42, -10, 8} +slices.Sort(smallInts) +fmt.Println(smallInts) +// Output: [-10 0 8 42] +``` + +## Database queries + +The web application relies on an Object-relational mapping (ORM) implementation provided by [SQLBoiler](https://github.com/volatiletech/sqlboiler) to simplify development. + +Some tutorials for SQLBoiler include: + +- [SQL in Go with SQLBoiler](https://thedevelopercafe.com/articles/sql-in-go-with-sqlboiler-ac8efc4c5cb8) +- [Introduction to SQLBoiler: Go framework for ORMs](https://blog.logrocket.com/introduction-sqlboiler-go-framework-orms/) + +--- + +Note, because I am lazy, the most of the descriptions below were generated using CoPilot AI. + +```go +import( + "github.com/Defacto2/server/internal/postgres/models" +) + +got, err := models.Files().One(ctx, db) +if err != nil { + panic(err) +} +fmt.Println("got record:", got.ID) +``` + +The above code snippet demonstrates how to query a single record from the `files` table. The `ctx` variable is a context.Context object and `db` is a *sql.DB object. + +#### Find a record by ID + +```go +f := &models.File{ID: 1} +found, err := models.FindFile(ctx, db, f.ID) +if err != nil { + log.DPanic(err) +} +fmt.Println("found record:", found.Filename) +``` + +The above code snippet demonstrates how to query a single record from the `files` table using a specific ID. + +#### Find records by ID + +```go +import( + "github.com/Defacto2/server/internal/postgres/models" + "github.com/volatiletech/sqlboiler/v4/queries/qm" +) + +found, err := models.Files(qm.Where("id = ?", 1)).One(ctx, db) +if err != nil { + log.DPanic(err) +} +fmt.Println("found record:", found.Filename) +``` + +The above code snippet also demonstrates how to query a single record from the `files` table using a specific ID. In this case, the query uses a query mod where clause. + +#### Check if a record exists + +```go +exists, err := models.FileExists(ctx, db, 1) +if err != nil { + log.DPanic(err) +} +fmt.Println("found again exists?:", exists) +``` + +The above code snippet demonstrates how to check if a record exists in the `files` table. + +#### Count records with deleted + +```go +count, err := models.Files(qm.WithDeleted()).Count(ctx, db) +if err != nil { + log.DPanic(err) +} +countPub, err := models.Files().Count(ctx, db) +if err != nil { + log.DPanic(err) +} +fmt.Println("total files vs total files that are public:", count, "vs", countPub) +``` + +The above code snippet demonstrates how to count the number of records in the `files` table. The first query counts all records, including those that have been soft-deleted. The second query counts only records that are not soft-deleted. + +#### Raw SQL queries + +```go +var users []*models.File +err := models.Files(qm.SQL(`select * from files;`)).Bind(ctx, db, &users) +if err != nil { + log.Fatal(err) +} +// range and print the results +fmt.Print("raw files:") +for i, user := range users { + if i != 0 { + fmt.Print(", ") + } + fmt.Print(user.Filename.String) +} +``` + +The above code snippet demonstrates how to execute a raw SQL query using SQLBoiler and bind the results to a slice of models. + +```go +var users []*models.File +rows, err := db.QueryContext(context.Background(), `select * from files;`) +if err != nil { + log.Fatal(err) +} +err = queries.Bind(rows, &users) +if err != nil { + log.Fatal(err) +} +rows.Close() +``` + +The above code snippet demonstrates how to execute a raw SQL query and bind the results to a slice of models. The `queries.Bind` function is provided by SQLBoiler. diff --git a/docs/readme_db.md b/docs/readme_db.md deleted file mode 100644 index 09e8245c..00000000 --- a/docs/readme_db.md +++ /dev/null @@ -1,126 +0,0 @@ -# Database readme - -Previous iterations of the Defacto2 web application relied on MySQL for its database. But for this 2023 application rewrite, the site will use PostgreSQL. - -Postgre is more strict about data types than MySQL. For example, inserting a string into a numeric column in Postgre will throw an error, whereas MySQL will convert the string to a number. Postgre has a more powerful query optimizer meaning queries often run faster with complex joins and subqueries. - -## Table and data changes to implement - -- Rename `Files` table to `Release` or `Releases` -- Create a `Release_tests` table with a selection of 20 read-only records. -- Rename `files.createdat` etc to `?_at` aka `create_at`. - -[PostgreSQL datatypes](https://www.postgresql.org/docs/current/datatype.html) - -`CITEXT` type for case-insensitive character strings. - -`files.filesize` should be converted to an `integer`, 4 bytes to permit a 2.147GB value. - -`files.id` should be converted to a `serial` type. - -### There is no performance improvement for `fixed-length`, padded character types, -Meaning strings can use `varchar`(n) or `text`. - -#### UUID - -`files.UUID` have be renamed from CFML style to the universal RFC-4122 syntax. - -This will require the modification of queries when dealing with `/files/[uuid|000|400]`. - -CFML is 35 characters, 8-4-4-16. -`xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx` - -RFC is 36 characters, 8-4-4-4-12. -`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` - -#### Store NFOs and texts - -We can store NFO and textfiles plus file_id.diz in database using the `bytea` hex-format, binary data type. It is more performant than the binary escape format. - -https://www.postgresql.org/docs/current/datatype-binary.html - -#### Full text seach types - -https://www.postgresql.org/docs/current/datatype-textsearch.html - -#### Files content relationship table - -Create a relationship files table that contains the filename content within of every archive release. - -We could also include columns containing size in bytes, sha256 hash, text body for full text searching. - -This would replace the `file_zip_content` column and also, create a CLI tool to scan the archives to fill out this data. For saftey and code maintenance, the tool would need to be a separate program from the web server application. - -## Migrate MySQL to PostgreSQL in a single, Docker command - -[This command relies on pgloader](https://pgloader.io/). - -> pgloader loads data into PostgreSQL and allows you to implement Continuous Migration from your current database to PostgreSQL. - -- `defacto2-inno` is the name of the MySQL database. -- `defacto2_ps` is the name of the PostgreSQL database. - -```sh -docker run --network host --rm -it dimitri/pgloader:latest \ - pgloader --verbose \ - mysql://root:example@127.0.0.1/defacto2-inno \ - pgsql://root:example@127.0.0.1/defacto2-ps -``` - -- [pgloader](https://pgloader.io/) -- [documentation](https://pgloader.readthedocs.io/en/latest/) -- [Docker hub](https://hub.docker.com/r/dimitri/pgloader/) -- [DigitalOcean how-to migrate](https://www.digitalocean.com/community/tutorials/how-to-migrate-mysql-database-to-postgres-using-pgloader) - -### Customize the migration - -A psloader migration is customized with a configuration file which might be more stable than using the commandline. - -[Migrating from MySQL to PostgreSQL](https://pgloader.readthedocs.io/en/latest/tutorial/tutorial.html#migrating-from-mysql-to-postgresql) - -`inno-to-ps.load` -```sql -LOAD DATABASE - FROM mysql://pgloader_my:mysql_password@mysql_server_ip/source_db?useSSL=true - INTO pgsql://pgloader_pg:postgresql_password@localhost/new_db - - WITH include drop, create tables - -ALTER SCHEMA 'source_db' RENAME TO 'public' -; -``` - -```sh -# run the migration -pgloader inno-to-ps.load -# test the migration -postgres psql -``` - -```sql -# SELECT * FROM files; -``` - - - -#### Database - -Firstly, set up the [Defacto2 PostgreSQL database](https://github.com/Defacto2/database-ps). - -```sh -# clone the database repository -cd ~ -git clone git@github.com:Defacto2/database-ps.git -cd ~/database-ps - -# migrate the Defacto2 data from MySQL to PostgreSQL -docker compose --profile migrater up - -# stop the running database by pressing CTRL+C -# cleanup the unnecessary volumes and containers -docker compose rm migrate mysql dbdump --stop -docker volume rm database-ps_tmpdump database-ps_tmpsql - -# restart the database to run in the background -docker compose up -d -``` diff --git a/docs/readme_go.md b/docs/readme_go.md deleted file mode 100644 index 6457870f..00000000 --- a/docs/readme_go.md +++ /dev/null @@ -1,108 +0,0 @@ -# Golang changes - -## v1.22 -https://go.dev/blog/go1.22 -https://tip.golang.org/doc/go1.22 - -New random package, [`math/rand/v2` package](https://pkg.go.dev/math/rand/v2). - -```go -alpha := []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"} -fmt.Println(alpha[rand.IntN(len(alpha))]) - -fmt.Println(rand.IntN(100)) -``` - -New type `Null` for [database/sql package](https://pkg.go.dev/database/sql#Null). - -```go -type Null[T any] struct { - V T - Valid bool -} -``` - -New [`go/version` package](https://pkg.go.dev/go/version) for version comparison. - -```go -func Compare(x, y string) int -func IsValid(x string) bool -func Lang(x string) string -``` - -## v1.21 -https://go.dev/blog/go1.21 -https://tip.golang.org/doc/go1.21 - - -New built-in functions: min, max, clear. -https://go.dev/ref/spec#Min_and_max -https://go.dev/ref/spec#Clear - - -New slog package for structured logging. -https://pkg.go.dev/log/slog - -```go -logger := slog.New(slog.NewJSONHandler(os.Stdout, nil)) -logger.Info("hello", "count", 3) - -slog.SetDefault(logger) - -logger2 := logger.With("url", r.URL) - -logger := slog.Default().With("id", systemID) -parserLogger := logger.WithGroup("parser") -parseInput(input, parserLogger) -``` - -New slices package for slice manipulation which are more efficient than `sort.`. -https://pkg.go.dev/slices - -```go - smallInts := []int8{0, 42, -10, 8} - slices.Sort(smallInts) - fmt.Println(smallInts) -``` - -New maps package. -https://pkg.go.dev/maps - -```go -maps.Clone -maps.Copy -maps.DeleteFunc -maps.Equal -maps.EqualFunc -``` - -New cmp package for comparing ordered values. -https://pkg.go.dev/cmp - -```go -cmp.Compare -cmp.Less -cmp.Or -cmp.Order -``` - -## v1.20 -https://go.dev/blog/go1.20 -https://tip.golang.org/doc/go1.20 - -New function `errors.Join` for joining multiple errors into a single error. -https://pkg.go.dev/errors#Join - -```go - err1 := errors.New("err1") - err2 := errors.New("err2") - err := errors.Join(err1, err2) - fmt.Println(err) - if errors.Is(err, err1) { - fmt.Println("err is err1") - } - if errors.Is(err, err2) { - fmt.Println("err is err2") - } -``` - diff --git a/docs/readme_handler.md b/docs/readme_handler.md deleted file mode 100644 index 626fb975..00000000 --- a/docs/readme_handler.md +++ /dev/null @@ -1,180 +0,0 @@ -# Handler readme - -The `handler/` directory is where the Echo router, controller and its settings exist. In a traditional model-view-controller framework, this `handler/` directory would contain the controller. - -This document lists possible new features and middleware that could be implemented into the existing Echo configuration. - ---- - -### Unit tests - -[This simple guide explains](https://echo.labstack.com/guide/testing/) how to test the fetching of a user by id from the database. If user is not found it returns 404 error with a message. - -[List all routes for testing and debugging](https://echo.labstack.com/guide/routing/#list-routes). - -[The list of middleware examples](https://github.com/labstack/echo/tree/master/middleware) contain `_test.go` test files. - ---- - -### CRUD - -User accounts example with create-read-update-delete, https://echo.labstack.com/cookbook/crud/ - ---- - -### Session ID - -Session middleware https://echo.labstack.com/middleware/session/ - -Request ID Middleware https://echo.labstack.com/middleware/request-id/ - ---- - -### Uploads - -[Upload a single file with parameters](https://echo.labstack.com/cookbook/file-upload/) example. - -[Upload multiple files with parameters](https://echo.labstack.com/docs/cookbook/file-upload#upload-multiple-files-with-parameters) example. - ---- - -### Timeouts - -#### DO NOT USE - -``` -// --------------------------------------------------------------------------------------------------------------- -// WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING -// WARNING: Timeout middleware causes more problems than it solves. -// WARNING: This middleware should be first middleware as it messes with request Writer and could cause data race if -// it is in other position -``` -[source code](https://github.com/labstack/echo/blob/master/middleware/timeout.go) - - -Timeout middleware for specific routes https://echo.labstack.com/middleware/timeout/ - ---- - -### Data validation - -Request and validate data sourced from FORMS, queries etc. -https://echo.labstack.com/guide/request/#custom-binder - ---- - -### Binding request data - -#### Data Sources - -Echo supports the following tags specifying data sources: - - query - query parameter - param - path parameter (also called route) - header - header parameter - json - request body. Uses builtin Go json package for unmarshalling. - xml - request body. Uses builtin Go xml package for unmarshalling. - form - form data. Values are taken from query and request body. Uses Go standard library form parsing. - -#### Data Types - -When decoding the request body, the following data types are supported as specified by the Content-Type header: - - application/json - application/xml - application/x-www-form-urlencoded - -When binding path parameter, query parameter, header, or form data, tags must be explicitly set on each struct field. However, JSON and XML binding is done on the struct field name if the tag is omitted. This is according to the behaviour of Go’s json package. - ---- - -### API `response` - -https://echo.labstack.com/guide/response/ - -#### OpenAPI - -- https://github.com/deepmap/oapi-codegen -- https://threedots.tech/post/serverless-cloud-run-firebase-modern-go-application/#public-http-api - ---- - -### Before and after hook responses - -To handle data outside of the templates. - -https://echo.labstack.com/guide/response/#hooks - ---- - -### Custom context - -HTTPS Server custom context that are used in handlers. - -https://echo.labstack.com/guide/context/ - ---- - -### HTTP/2 Cleartext Server - -https://echo.labstack.com/cookbook/http2/ - ---- - -### CORS - -https://echo.labstack.com/middleware/cors/ - -https://echo.labstack.com/cookbook/cors/ - ---- - -### CSRF Middleware - -https://echo.labstack.com/middleware/csrf/ - ---- - -### Sub-domains - -http://html3.defacto2.net ? - -https://echo.labstack.com/cookbook/subdomains/ - ---- - -## Proxy or no proxy? - -#### IP config options - -https://echo.labstack.com/guide/ip-address/ - -## Case 1. With no proxy - -If you put no proxy (e.g.: directory facing to the internet), all you need to (and have to) see is IP address from network layer. Any HTTP header is untrustable because the clients have full control what headers to be set. - -In this case, use echo.ExtractIPDirect(). - -## Case 2. With proxies using X-Forwarded-For header - -`X-Forwared-For` (XFF) is the popular header to relay clients’ IP addresses. At each hop on the proxies, they append the request IP address at the end of the header. - -## Case 3. With proxies using X-Real-IP header - -`X-Real-IP` is another HTTP header to relay clients’ IP addresses, but it carries only one address unlike XFF. - ---- - -## Event-driven applications - -The intention is to enable two or more binaries to communicate with each other, ie `df2` and `server`. When `df2` updates, it could tell the `server` application to refresh some cached data stores such as the group statistics. - -[Introducing Watermill - Go event-driven applications library](https://threedots.tech/post/introducing-watermill/) - -https://watermill.io/ - -https://github.com/ThreeDotsLabs/watermill - -https://github.com/ThreeDotsLabs/watermill/tree/master/_examples - ---- \ No newline at end of file diff --git a/docs/readme_model.md b/docs/readme_model.md deleted file mode 100644 index ae2a9f2f..00000000 --- a/docs/readme_model.md +++ /dev/null @@ -1,114 +0,0 @@ -# Model readme - -The model directory is where the SQLBoiler QueryMods and building blocks exist. These blocks are interchangeable between different SQL database applications. - -### [SQLBoiler readme](https://github.com/volatiletech/sqlboiler) - -### Example tutorials - -[Handling HTTP request in Go Echo framework](https://blog.boatswain.io/post/handling-http-request-in-go-echo-framework-1/) - -[SQL in Go with SQLBoiler](https://thedevelopercafe.com/articles/sql-in-go-with-sqlboiler-ac8efc4c5cb8) - -[Introduction to SQLBoiler: Go framework for ORMs](https://blog.logrocket.com/introduction-sqlboiler-go-framework-orms/) - ---- - -### SQLBoiler example - -```go -got, err := models.Files().One(ctx, db) -if err != nil { - log.DPanic(err) -} -fmt.Println("got record:", got.ID) - -f := &models.File{ID: 1} -found, err := models.FindFile(ctx, db, f.ID) -if err != nil { - log.DPanic(err) -} -fmt.Println("found record:", found.Filename) - -foundAgain, err := models.Files(qm.Where("id = ?", 1)).One(ctx, db) -if err != nil { - log.DPanic(err) -} -fmt.Println("found again record:", foundAgain.Filename) - -exists, err := models.FileExists(ctx, db, foundAgain.ID) -if err != nil { - log.DPanic(err) -} -fmt.Println("found again exists?:", exists) - -count, err := models.Files(qm.WithDeleted()).Count(ctx, db) -if err != nil { - log.DPanic(err) -} -countPub, err := models.Files().Count(ctx, db) -if err != nil { - log.DPanic(err) -} -fmt.Println("total files vs total files that are public:", count, "vs", countPub) - -var users []*models.File -rows, err := db.QueryContext(context.Background(), `select * from files;`) -if err != nil { - log.Fatal(err) -} -err = queries.Bind(rows, &users) -if err != nil { - log.Fatal(err) -} -rows.Close() - -users = nil -err = models.Files(qm.SQL(`select * from files;`)).Bind(ctx, db, &users) -if err != nil { - log.Fatal(err) -} -fmt.Print("raw files:") -for i, u := range users { - if i != 0 { - fmt.Print(", ") - } - fmt.Print(u.Filename.String) -} -fmt.Println() -``` - -### Extending SQL Boiler's generated models - -https://github.com/volatiletech/sqlboiler#extending-generated-models - - -```go -# method 1: simple funcs - -// Package modext is for SQLBoiler helper methods -package modext - -// UserFirstTimeSetup is an extension of the user model. -func UserFirstTimeSetup(ctx context.Context, db *sql.DB, u *models.User) error { ... } - -# calling code - -user, err := Users().One(ctx, db) -// elided error check - -err = modext.UserFirstTimeSetup(ctx, db, user) -// elided error check -``` - -### New clone table for testing data. - -Create a new PostgreSQL, Files DB for unit testing. - -`defacto2-tests` containing `files`, `groupnames` - -```sh -sudo -u postgres -- createuser --createdb --pwprompt start -createdb --username start --password --owner start gettingstarted -psql --username start --password gettingstarted < schema.sql -``` diff --git a/docs/todo.md b/docs/todo.md index 3ba1103a..f3e0d6e5 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -1,56 +1,28 @@ # TODOs and tasks - * (star) prefix indicates a *low priority* task. + * (star) __*__ prefix indicates a *low priority* task. + * (question) __?__ prefix indicates an *idea* or *doubtful*. -### CLI +### Terminal commands and flags -- [ ] * Command to cleanup the database? -- [ ] * Command to reindex the database? Both to erase and rebuild the indexes. +- [ ] *? Command to clean up the database and remove all orphaned records. +- [ ] *? Command to reindex the database, both to erase and rebuild the indexes. -### Files +### Files and assets -- [ ] Handle magazines with the file editor. -- [ ] Create a htmx live classifications page for editors, using the adv uploader ` fields. -### Layout +### Menus and layout -- [ ] Create a menu link to DO referal page, or/and add a link to the thanks page. -- [ ] Create a locked menu option to search by database ID or UUID or ~~URL~~. +- [ ] Create a menu link to DigitalOcean referal page, [or/and] add a link to the thanks page. +- [ ] Create a locked menu option to search the database by file ID or UUID or ~~URL~~. ### Database -- [ ] Create DB fix to detect and rebadge msdos and windows trainers. -- [ ] Create PostgreSQL *indexes* with case-sensitive strings. - https://wirekat.com/optimizing-sql-based-on-postgresql/ - | Partial Indexes: Use partial indexes when you only need to index a subset of rows, such as CREATE INDEX ON orders (order_date) WHERE status = 'SHIPPED'; - | Over-Indexing: Creating too many indexes can slow down write operations, as each index needs to be updated on INSERT, UPDATE, or DELETE. - | Index Maintenance: Rebuild indexes periodically to deal with bloat using REINDEX. - | Indexing Join Columns: Index columns that are used in JOIN conditions to improve join performance. - > `combineGroup` and `(r Role) Distinct()` +- [ ] Create a DB fix to detect and rebadge msdos and windows trainers. - [ ] `OrderBy` Name/Count /html3/groups? https://pkg.go.dev/sort#example-package-SortKeys -- [ ] [model.Files.ListUpdates], rename the PSQL column from "updated_at" to "date_updated". ### Backend -- [ ] * [Implememnt a sheduling library for Go](https://github.com/reugn/go-quartz) - -#### Support Unicode slug URLs as currently the regex removes all non alphanumeric chars. - -```go -/* -Error: Not equal: - expected: "Mooñpeople" - actual : "Moopeople" - - use utf8 lib to detect extended chars? -*/ -``` - -### URL /v/xxx pattern tests. - -- [X] detect if the file contains /r/n or /n and replace with /n only. - example, http://localhost:1323/v/af18f9b -- [X] detect if the file uses cp437 or unicode and convert to utf8. - example, http://localhost:1323/v/b01de5b - http://localhost:1323/v/b521c83 - http://localhost:1323/v/b8297cf +- [ ] *? Implememnt a [scheduling library for Go](https://github.com/reugn/go-quartz) diff --git a/internal/tags/strings.go b/internal/tags/strings.go index 24420347..859fc6ad 100644 --- a/internal/tags/strings.go +++ b/internal/tags/strings.go @@ -27,6 +27,7 @@ func (t Tag) String() string { } func URIs() URIS { + return URIS{ Announcement: "announcements", ANSIEditor: "ansieditor",