monorepo (#4074)

* - convert project to use pnpm instead of npm - docs: mark e2e testing section as "defect" - remove vscode lint task * remove unused file * remove testcafe stuff * move bulk of source files, does not work yet * more progress like adjusting paths, making new packages and updating docs * ts: change all targets to es2018 * set min pnpm and nodejs versions in package.json * add check:types script * use pnpm workspace catalogs for shared dependencies * add scripts to change the core thats used * fix fmt * script to update/sync versions for targets * replace remaining references to electron from runtime interface: - `Electron.OpenDialogOptions` -> `RuntimeOpenDialogOptions` - qelectron.app.getPath()` -> `RuntimeAppPath` * fix lint of shared import in frontend * disable drag and drop files for now It is easier to fix later when we can test it * on paste: remove usage of `File.path` which was not set in some of the cases. * add missing debounce dependency to target-electron * fix lint of shared import in target-electron * global definition for electron-runtime * fix fmt * remove last direct usage of electron from frontend. move Deltachat jsonrpc transport to runtime-interface implementation * bundle backend code (targe-electron), move most dependencies to dev dependencies because they are now bundled * proper native sourcemap support during development for main process * mention `NODE_OPTIONS=--enable-source-maps` in docs * document `TEST_DIR` * fix loading locales in electron main process for devleopment Also add `DELTACHAT_LOCALE_DIR`env var to allow replacement of the localse folder. useful for debuging but you could also make a tool for users to live edit translations in a released version of deltachat together with the `--translation-watch` flag. * fix fmt * add the idea for end user translation tool to docs in a footnote * make build info part of the runtime and its generation part of the target-electron build script * rm used yerpc import in backend-com * move non shared types from shared to where they belong. Also improved some comments. * sort scripts by whether they work yet * progress on making frontend build (ts and scss so far) * fix path in comment * - sort static files into to their projects - new location for themes html-dist - make build themes script a bit faster and simpler * make copy fail with exit code 1 if dir is not found (unless it is started in watch mode) * make it run again (styles are broken but basics and logic work) * reintroduce dev script as alias to dev:electron to make transition easier for devs * fix blueprint import (bundle it) * fix images and icons * fix loading themes * update theming docs * remove dead code * use correct log level when logging to console in main process * make drag and drop work again this could have fixed the following issue #3595 as a side effect, if not then that issue is not fixable, altleast unless we start interacting with the OS Apis directly in a native module or by contributing to electron. * harden electron functions, only runtime can use them now * harden runtime by deleting the reference on window (`window.r`) after the first use. For development it is now accessible at `exp.runtime` but only in --devmode like `exp.rpc` * update_core docs: update the start command for electron * fix eslint issues and fix formatting * improve ./bin/test_for_missing_translations.sh script it's now more correct, helpful and faster * update ./bin/topEvil.sh script * scripts to update and check package versions * update prettierignore * update ./bin/update_background_thumbnails.sh script * fmt language list * update theme builder script to use compileString instead of renderSync, because the latter is deprecated * fix that composer was monospace font * tests for packages/shared * make migration test work * add testing to ci * add test-and-check script * make watch scripts work * docs: add hint about completion in terminal for pnpm * make packaging work again * update ci and docs * make `check:log-conventions` part of checks * update dev docs * update more pnpm commands * delete old package.json * update file tree in docs * ci: fix building preview * update the versions (for/from main branch rebase) * add changelog * runtime: also use catalog: dependency on @deltachat/jsonrpc-client to fix tests * test building in ci and fix ci test * ci: fix dev-builds * add comment to `previewBuild` variable in gen-electron-builder-config.js * update target versions * fix preview build ci * fix devbuild * fix tray and app icon * other appid for preview build on mac * fix readme commands * Fix if Contact has long bio, then shared chats are hidden (#4093) * set min height for shared chats in profile view fixes #4092 * more clear varname * refactor styles a tiny bit to make status text scroll if there is little space * remove workaround from #4068, can't reproduce the bug of it anymore * fix that you can not click header button in dialog when they are on top of the navbar also see #4018 * add changelog entries * fix fmt * update versions after rebase to match main branch: - deltachat-core (stdio-server and jsonrpc-client) 1.142.10 - electron-notarize 2.4 * fix chatlist image thumbnails (#4101) * fix chatlist image thumbnails * changelog entry * update core after rebase * fix changelog
deltachat · Sep 9, 2024 · a5c0243 · a5c0243
1 parent 5018d9d
commit a5c0243
Show file tree

Hide file tree

Showing 535 changed files with 8,833 additions and 10,388 deletions.
diff --git a/.eslintignore b/.eslintignore
@@ -23,3 +23,6 @@ test
 
 index.js
 .eslintrc.js
+
+packages/shared/ts-compiled-for-tests
+packages/target-electron/tests/compiled
diff --git a/.github/workflows/basic-tests.yml b/.github/workflows/basic-tests.yml
@@ -1,4 +1,4 @@
-name: Basic Code Checks
+name: Basic Code Checks & Tests
 
 on:
   push:
@@ -12,7 +12,7 @@ on:
       - '.vscode/**'
 
 jobs:
-  test:
+  check:
     name: Code Validation
     runs-on: ubuntu-latest
 
@@ -22,18 +22,29 @@ jobs:
       uses: actions/setup-node@v1
       with:
         node-version: 20.x
-    - name: npm ci
-      run: |
-        npm ci
-    - name: preperation (generate code)
+    - name: pnpm install
       run: |
-        npm run build:shared:version
+        npm i -g pnpm
+        pnpm install --frozen-lockfile
     - name: Run all checks
       run: |
-        npm run check
-    - name: Check for misplaced console.log statements
+        pnpm check
+  test:
+    name: Code Tests
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Use Node.js 20.x
+      uses: actions/setup-node@v1
+      with:
+        node-version: 20.x
+    - name: pnpm install
       run: |
-        npm run test-log-conventions
-    - name: Misc tests
+        npm i -g pnpm
+        pnpm install --frozen-lockfile
+    - name: Build
+      run: pnpm build:electron
+    - name: Run all checks
       run: |
-        npm run test-misc
+        pnpm test
diff --git a/.github/workflows/build-preview.yml b/.github/workflows/build-preview.yml
@@ -34,19 +34,24 @@ jobs:
       run: |
         node ./bin/github-actions/devbuild.js
     # Building JS
-    - name: npm install, build
+    - name: pnpm install, build
       run: |
-        npm install
-        npm run build
-        npm run pack:generate_config
+        npm i -g pnpm
+        pnpm install --frozen-lockfile
+        cd packages/target-electron
+        pnpm build
+        pnpm run pack:generate_config
+        pnpm run pack:patch-node-modules
     # Packaging and processing
     - name: LINUX electron builder
       if: runner.os == 'Linux'
       run: |
         export DEBUG=electron-builder
-        npm exec -- electron-builder --publish never --linux AppImage
+        cd packages/target-electron
+        pnpm electron-builder --publish never --linux AppImage
     - name: LINUX renaming / upload Preperation
       if: runner.os == 'Linux'
+      working-directory: packages/target-electron
       run: |
         cd dist
         mkdir preview
@@ -56,16 +61,19 @@ jobs:
     - name: MAC electron builder
       if: runner.os == 'macOS'
       run: |
+        cd packages/target-electron
         echo "module.exports=()=>Promise.resolve()" > stub.cjs
         export CSC_IDENTITY_AUTO_DISCOVERY=false
         export DEBUG=electron-builder
-        npm i dmg-license
-        npm i --cpu x64 --os darwin
-        npm i --cpu arm64 --os darwin
+        pnpm i dmg-license
+        # all archs are enabled, because only installing some is not supported yet by pnpm: https://github.com/pnpm/pnpm/issues/7510
+        # pnpm i --cpu x64 --os darwin
+        # pnpm i --cpu arm64 --os darwin
         ls node_modules/@deltachat
-        npm exec -- electron-builder --publish never --mac mas dmg -c.mac.identity=null --universal --config.afterSign="stub.cjs"
+        pnpm electron-builder --publish never --mac mas dmg -c.mac.identity=null --universal --config.afterSign="stub.cjs"
     - name: MAC renaming / upload Preperation
       if: runner.os == 'macOS'
+      working-directory: packages/target-electron
       run: |
         ls -lah dist
         mkdir -p dist/preview
@@ -77,12 +85,13 @@ jobs:
     - name: WINDOWS electron builder
       if: runner.os == 'Windows'
       run: |
+        cd packages/target-electron
         set DEBUG=electron-builder
-        npm exec -- electron-builder --win portable --config.artifactName="deltachat-desktop-${{ steps.prepare.outputs.prid }}.portable.exe"
+        pnpm electron-builder --win portable --config.artifactName="deltachat-desktop-${{ steps.prepare.outputs.prid }}.portable.exe"
     - name: WINDOWS renaming / upload Preperation
       if: runner.os == 'Windows'
+      working-directory: packages/target-electron/dist
       run: |
-        cd dist
         dir
         mkdir preview
         move deltachat-desktop-${{ steps.prepare.outputs.prid }}.portable.exe preview/deltachat-desktop-${{ steps.prepare.outputs.prid }}.portable.exe
@@ -95,7 +104,7 @@ jobs:
       run: |
         echo -e "${{ secrets.KEY }}" >__TEMP_INPUT_KEY_FILE
         chmod 600 __TEMP_INPUT_KEY_FILE
-        scp -o StrictHostKeyChecking=no -v -i __TEMP_INPUT_KEY_FILE -P "22" -r dist/preview/* "${{ secrets.USERNAME }}"@"download.delta.chat":"/var/www/html/download/desktop/preview/"
+        scp -o StrictHostKeyChecking=no -v -i __TEMP_INPUT_KEY_FILE -P "22" -r packages/target-electron/dist/preview/* "${{ secrets.USERNAME }}"@"download.delta.chat":"/var/www/html/download/desktop/preview/"
       continue-on-error: true
     - name: "Post links to details"
       if: steps.upload.outcome == 'success'
@@ -108,4 +117,4 @@ jobs:
       uses: actions/upload-artifact@master
       with:
         name: ${{ matrix.os }} output
-        path: dist/preview/
+        path: packages/target-electron/dist/preview/
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -10,7 +10,7 @@ on:
 
 jobs:
   test:
-    name: Tests and E2E Tests
+    name: Tests #and E2E Tests
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
@@ -22,20 +22,21 @@ jobs:
       uses: actions/setup-node@v1
       with:
         node-version: 20.x
-    - name: npm ci
+    - name: install dependencies
       run: |
-        npm ci
-    - name: npm run build
+        npm i -g pnpm
+        pnpm install --frozen-lockfile
+    - name: build
       run: |
-        npm run build
-    - name: npm test-unit
+        pnpm run build
+    - name: test
       run: |
-        npm run test-unit
-    - uses: DevExpress/testcafe-action@latest
-      with:
-        version: "1.15.0"
-        args: "--skip-js-errors electron:. ./test/testcafe/test.js ./test/testcafe/messagelist_tests.ts"
-      env:
-        DCC_NEW_TMP_EMAIL: ${{secrets.DCC_NEW_TMP_EMAIL}}
-        NODE_ENV: test
-        TEST_DIR: .test_tmp_data
+        pnpm run test
+    # - uses: DevExpress/testcafe-action@latest
+    #   with:
+    #     version: "1.15.0"
+    #     args: "--skip-js-errors electron:. ./test/testcafe/test.js ./test/testcafe/messagelist_tests.ts"
+    #   env:
+    #     DCC_NEW_TMP_EMAIL: ${{secrets.DCC_NEW_TMP_EMAIL}}
+    #     NODE_ENV: test
+    #     TEST_DIR: .test_tmp_data
diff --git a/.gitignore b/.gitignore
@@ -77,10 +77,6 @@ static/conversations.css
 # portable config folders
 DeltaChatData
 
-static/fallback-theme.css
-themes/*.css
-
-src/shared/build-info.ts
 # ignore provisioning file
 embedded.provisionprofile
 
@@ -89,9 +85,6 @@ stub.js
 # message list e2e test
 .test_tmp_data
 
-# pnpm lockfile
-pnpm-lock.yaml
-
 deltachat-rpc-server-aarch64-macos
 .direnv
 .envrc

diff --git a/.npmrc b/.npmrc
@@ -0,0 +1,2 @@
+# this allows us to use env vars in scripts in package.json crossplatform
+shell-emulator=true
diff --git a/.prettierignore b/.prettierignore
@@ -1,4 +1,7 @@
 CONTRIBUTORS.md
 CHANGELOG.md
 node_modules
-src/shared/build-info.ts
+
+packages/target-electron/bundle_out
+packages/target-electron/tests/compiled
+packages/shared/ts-compiled-for-tests
diff --git a/.prettierrc.yml b/.prettierrc.yml
@@ -6,3 +6,11 @@ singleQuote: true
 jsxSingleQuote: true
 endOfLine: auto
 arrowParens: avoid
+
+overrides:
+- files: "*.json5"
+  options:
+    singleQuote: false
+    quoteProps: preserve
+    trailingComma: none
+    parser: json5
diff --git a/.testcafe-electron-rc b/.testcafe-electron-rc
diff --git a/.vscode/tasks.json b/.vscode/tasks.json
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,8 +5,21 @@
 ### Added
 
 ### Changed
-
-### Fixed
+- dev: transformed repo into monorepo
+- dev: switched from `npm` to `pnpm`
+- dev: esbuild bundling for electron main process js (+minification for releases)
+- changed implementation for accepting dropped in files, use browser apis instead of electron specific hack.
+- dev: improved `./bin/test_for_missing_translations.sh` script: It is now more correct, helpful and faster
+
+### Fixed
+- fix that you can not click header button in dialog when they are on top of the navbar #4093
+- fix if Contact has long bio/signature, then shared chats were hidden #4093
+- dev: proper native source-map support during development for main process
+- dev: use correct log level when logging to console in main process
+- security: harden electron_functions, only runtime can use them now
+- security: harden runtime interface by deleting the reference on window (`window.r`) after the first use. For development it is now accessible at `exp.runtime` but only in `--devmode` like `exp.rpc`
+- dev: update `./bin/update_background_thumbnails.sh` script
+- fix chatlist image thumbnails #4101
 
 <a id="1_46_8"></a>
 
@@ -20,6 +33,8 @@
   - Display `Config::MdnsEnabled` as true by default.
 
 ### Fixed
+- fix that you can not click header button in dialog when they are on top of the navbar #4093
+- fix if Contact has long bio/signature, then shared chats were hidden #4093
 - fix newlines in messages with WebXDC attachments #4079
 - being unable to delete a nonfunctional account imported from ArcaneChat #4104
 - Ctrl/Cmd+Q (also File->Quit) now should properly close the app when focus is on main window
@@ -34,6 +49,7 @@
 - Update `@deltachat/stdio-rpc-server` and `deltachat/jsonrpc-client` to `1.142.11`
   - Set backward verification when observing `vc-contact-confirm` or `vg-member-added`
 
+
 <a id="1_46_6"></a>
 
 ## [1.46.6] - 2024-08-29

diff --git a/README.md b/README.md
@@ -142,16 +142,21 @@ You can find the downloads for windows on <https://get.delta.chat>, though we re
 $ git clone https://github.com/deltachat/deltachat-desktop.git
 $ cd deltachat-desktop
 
+# Install pnpm
+$ npm i -g pnpm
+
 # Install dependencies
-$ npm install
+$ pnpm install
 
 # Build the app (only needed on the first time or if the code was changed)
-$ npm run build
+$ pnpm -w build:electron
 
 # Start the application:
-$ npm start
+$ pnpm -w start:electron
 ```
 
+> `-w` means workspace root package, with this you don't need to have your current working directory at the repo-root to run those scripts.
+
 For development with local deltachat-core read the [docs](docs/UPDATE_CORE.md)
 
 ### Troubleshooting <a id="troubleshooting"></a>

diff --git a/RELEASE.md b/RELEASE.md
@@ -61,7 +61,7 @@
 1. Make sure no "blocking" bugs are in the issue tracker. If there's any, try
    to solve them first:
    <https://github.com/deltachat/deltachat-desktop/issues?q=is%3Aopen+is%3Aissue+label%3Ablocker>
-2. Pull translations via `npm run translations-update` in a separate PR
+2. Pull translations via `pnpm -w translations:update` in a separate PR
 3. Update the local help files if necessary in a separate PR:
    <https://github.com/deltachat/deltachat-pages/blob/master/tools/create-local-help.py>
 4. If you've updated any translations or local help files, make sure that the
@@ -75,7 +75,7 @@
    section with your new version code)
 3. Do not forget to update the tag links at the end of the `CHANGELOG.md` file!
 4. Change `version` field in `package.json` to `X.Y.Z`
-5. Run `npm install` to update `package-lock.json`
+5. Run `pnpm -w update:target-versions` to update versions in the other packages
 6. Open a PR for your branch and get it reviewed.
 7. As soon as your PR is approved merge it to `main`
 8. After the PR is merged, checkout the latest version on `main`. Tag the latest commit

diff --git a/_locales/_languages.json b/_locales/_languages.json
@@ -4,12 +4,12 @@
   "bg": "Български език",
   "bqi": "Luri Bakhtiari",
   "ca": "Català",
-  "cs": "Čeština",
   "ckb": "کوردی",
+  "cs": "Čeština",
   "da": "Dansk",
   "de": "Deutsch",
-  "en": "English",
   "el": "ελληνικά",
+  "en": "English",
   "eo": "Esperanto",
   "es": "Español",
   "eu": "Euskara",
@@ -44,4 +44,4 @@
   "vi": "Tiếng Việt",
   "zh_CN": "简体中文",
   "zh_TW": "繁體中文"
-}
+}