Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve / fix naming in NL server #4263

Merged
merged 13 commits into from
May 24, 2024
Merged

Improve / fix naming in NL server #4263

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
ebd7aa2
update nl server resource/object naming
shifucun May 22, 2024
d5b5d73
Merge branch 'master' of https://github.com/datacommonsorg/website in…
shifucun May 22, 2024
3f96fdd
Merge branch 'master' of https://github.com/datacommonsorg/website in…
shifucun May 22, 2024
976995c
Merge branch 'master' of https://github.com/datacommonsorg/website in…
shifucun May 22, 2024
e2085ff
more fix
shifucun May 22, 2024
e23a892
sync
shifucun May 23, 2024
3406e63
Merge branch 'master' of https://github.com/datacommonsorg/website in…
shifucun May 23, 2024
cd50fe6
more
shifucun May 23, 2024
75f68a3
Merge branch 'master' of https://github.com/datacommonsorg/website in…
shifucun May 24, 2024
39a4e71
add back model.yaml
shifucun May 24, 2024
24f2f93
fix tests
shifucun May 24, 2024
fd22fe5
resolve comments
shifucun May 24, 2024
c1a7d6f
wb
shifucun May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build/website_cron_testing/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
absl-py==1.4.0
requests==2.31.0
selenium==4.9.0
selenium==4.21.0
webdriver-manager==4.0.0
deepdiff==6.7.1
google-cloud-storage==2.15.0
Expand Down
3 changes: 1 addition & 2 deletions deploy/helm_charts/dc_website/templates/config_maps.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,7 @@ metadata:
namespace: {{ .Values.namespace.name }}
data:
embeddings.yaml: {{ required "NL embeddings file is required" .Values.nl.embeddings | quote }}
models.yaml: {{ required "NL models file is required" .Values.nl.models | quote }}
embeddings_spec.json: {{ .Values.nl.embeddingsSpec | toJson | quote }}
env.yaml: {{ .Values.nl.env | quote }}
{{- end }}

{{- if .Values.website.redis.enabled }}
Expand Down
12 changes: 5 additions & 7 deletions deploy/helm_charts/dc_website/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,15 +133,13 @@ serviceGroups:
nl:
enabled: false
embeddings:
models:
memory: "2G"
workers: 1
embeddingsSpec:
defaultIndex: ""
enabledIndexes: []
vertexAIModels:
enableReranking: false

env:
default_indexes: []
enabled_indexes: []
vertex_ai_models: {}
enable_reranking: false

###############################################################################
# Config for Stat Var Groups which is shared between Website and Mixer
Expand Down
33 changes: 16 additions & 17 deletions deploy/helm_charts/envs/autopush.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,22 +44,21 @@ serviceAccount:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "base_uae_mem"
enabledIndexes:
[
"base_uae_mem",
"bio_ft",
"medium_ft",
"medium_lance_ft",
"medium_vertex_ft",
"medium_vertex_mistral",
"sdg_ft",
"undata_ft",
"undata_ilo_ft",
"undata_dev_ft",
]
vertexAIModels:
env:
default_indexes:
- base_uae_mem
enabled_indexes:
- base_uae_mem
- bio_ft
- medium_ft
- medium_lance_ft
- medium_vertex_ft
- medium_vertex_mistral
- sdg_ft
- undata_ft
- undata_ilo_ft
- undata_dev_ft
vertex_ai_models:
dc-all-minilm-l6-v2-model:
project_id: datcom-website-dev
location: us-central1
Expand All @@ -80,7 +79,7 @@ nl:
project_id: datcom-website-dev
location: us-central1
prediction_endpoint_id: "284894457873039360"
enableReranking: true
enable_reranking: true

serviceGroups:
recon: null
Expand Down
10 changes: 7 additions & 3 deletions deploy/helm_charts/envs/bard.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,13 @@ nl:
enabled: true
memory: "10G"
workers: 3
embeddingsSpec:
defaultIndex: "medium_ft"
enabledIndexes: ["medium_ft", "sdg_ft", "undata_ft"]
env:
default_indexes:
- medium_ft
enabled_indexes:
- medium_ft
- sdg_ft
- undata_ft

nodejs:
enabled: true
Expand Down
16 changes: 8 additions & 8 deletions deploy/helm_charts/envs/biomedical.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ serviceGroups:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "bio_ft"
enabledIndexes: [
"bio_ft"
"medium_ft",
"sdg_ft",
"undata_ft",
]
env:
default_indexes:
- bio_ft
enabled_indexes:
- bio_ft
- medium_ft
- sdg_ft
- undata_ft
8 changes: 5 additions & 3 deletions deploy/helm_charts/envs/climate_trace.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,11 @@ serviceAccount:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "medium_ft"
enabledIndexes: ["medium_ft", "sdg_ft", "undata_ft"]
env:
default_indexes:
- medium_ft
enabled_indexes:
- medium_ft

serviceGroups:
recon: null
Expand Down
33 changes: 16 additions & 17 deletions deploy/helm_charts/envs/dev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,22 +44,21 @@ serviceGroups:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "base_uae_mem"
enabledIndexes:
[
"base_uae_mem",
"bio_ft",
"medium_ft",
"medium_lance_ft",
"medium_vertex_ft",
"medium_vertex_mistral",
"sdg_ft",
"undata_ft",
"undata_ilo_ft",
"undata_dev_ft",
]
vertexAIModels:
env:
default_indexes:
- base_uae_mem
enabled_indexes:
- base_uae_mem
- bio_ft
- medium_ft
- medium_lance_ft
- medium_vertex_ft
- medium_vertex_mistral
- sdg_ft
- undata_ft
- undata_ilo_ft
- undata_dev_ft
vertex_ai_models:
dc-all-minilm-l6-v2-model:
project_id: datcom-website-dev
location: us-central1
Expand All @@ -80,7 +79,7 @@ nl:
project_id: datcom-website-dev
location: us-central1
prediction_endpoint_id: "284894457873039360"
enableReranking: true
enable_reranking: true

nodejs:
enabled: true
Expand Down
8 changes: 5 additions & 3 deletions deploy/helm_charts/envs/internal.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,11 @@ ingress:
certName: website-ssl-certificate
nl:
enabled: true
embeddingsSpec:
defaultIndex: "medium_ft"
enabledIndexes: ["medium_ft", "sdg_ft", "undata_ft"]
env:
default_indexes:
- medium_ft
enabled_indexes:
- medium_ft

serviceGroups:
recon: null
Expand Down
8 changes: 5 additions & 3 deletions deploy/helm_charts/envs/magic_eye.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,11 @@ ingress:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "medium_ft"
enabledIndexes: ["medium_ft", "sdg_ft", "undata_ft"]
env:
default_indexes:
- medium_ft
enabled_indexes:
- medium_ft

serviceGroups:
recon: null
Expand Down
15 changes: 10 additions & 5 deletions deploy/helm_charts/envs/prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,20 @@ serviceAccount:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "base_uae_mem"
enabledIndexes: ["base_uae_mem", "medium_ft", "sdg_ft", "undata_ft"]
vertexAIModels:
env:
default_indexes:
- base_uae_mem
enabled_indexes:
- base_uae_mem
- medium_ft
- sdg_ft
- undata_ft
vertex_ai_models:
uae-large-v1-model:
project_id: datcom-nl
location: us-central1
prediction_endpoint_id: "430892009855647744"
enableReranking: false
enable_reranking: false

serviceGroups:
recon: null
Expand Down
15 changes: 10 additions & 5 deletions deploy/helm_charts/envs/staging.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,20 @@ serviceAccount:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "base_uae_mem"
enabledIndexes: ["base_uae_mem", "medium_ft", "sdg_ft", "undata_ft"]
vertexAIModels:
env:
default_indexes:
- base_uae_mem
enabled_indexes:
- base_uae_mem
- medium_ft
- sdg_ft
- undata_ft
vertex_ai_models:
uae-large-v1-model:
project_id: datcom-nl
location: us-central1
prediction_endpoint_id: "8435477397553283072"
enableReranking: false
enable_reranking: false

serviceGroups:
recon: null
Expand Down
10 changes: 7 additions & 3 deletions deploy/helm_charts/envs/unsdg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,13 @@ ingress:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "sdg_ft"
enabledIndexes: ["medium_ft", "sdg_ft", "undata_ft"]
env:
default_indexes:
- sdg_ft
enabled_indexes:
- medium_ft
- sdg_ft
- undata_ft

serviceGroups:
recon: null
Expand Down
20 changes: 9 additions & 11 deletions deploy/helm_charts/envs/unsdg_staging.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,17 +37,15 @@ mixer:

nl:
enabled: true
embeddingsSpec:
defaultIndex: "sdg_ft"
enabledIndexes:
[
"bio_ft",
"medium_ft",
"sdg_ft",
"undata_ft",
"undata_ilo_ft",
"undata_dev_ft",
]
env:
default_indexes:
- sdg_ft
enabled_indexes:
- medium_ft
- sdg_ft
- undata_ft
- undata_ilo_ft
- undata_dev_ft

serviceGroups:
recon: null
Expand Down
8 changes: 5 additions & 3 deletions deploy/helm_charts/envs/worldbank.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,11 @@ serviceGroups:
memoryLimit: "8G"
nl:
enabled: true
embeddingsSpec:
defaultIndex: "medium_ft"
enabledIndexes: ["medium_ft", "sdg_ft", "undata_ft"]
env:
default_indexes:
- medium_ft
enabled_indexes:
- medium_ft
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intend to remove others?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intended to only keep medium_ft for these instances. For world bank, do you think we should keep sdg and undata?

Copy link
Contributor

@pradh pradh May 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since worldbank is part of un, maybe keep it there, could come handy for any demos, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok


svg:
blocklistFile: ["dc/g/Uncategorized", "oecd/g/OECD"]
Loading
Loading