Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add notebook for VAIS automatic URL recrawl #161

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat: Add notebook for VAIS automatic URL recrawl #161

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f450fce
feat: Add notebook for VAIS automatic URL recrawl
hossein-mansour Jan 4, 2025
40dd793
Update README.md
hossein-mansour Jan 4, 2025
ec441c8
Update mkdocs.yml
hossein-mansour Jan 4, 2025
4cf5074
Update README.md
hossein-mansour Jan 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
- content.footnote.tooltips
- content.tabs.link
- content.tooltips
- header.autohide

Check warning on line 28 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`autohide` is not a recognized word. (unrecognized-spelling)
- navigation.expand
- navigation.footer
- navigation.indexes
Expand Down Expand Up @@ -124,22 +124,22 @@
- md_in_html
- toc:
permalink: true
- pymdownx.arithmatex:

Check warning on line 127 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`pymdownx` is not a recognized word. (unrecognized-spelling)
generic: true
- pymdownx.betterem:

Check warning on line 129 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`betterem` is not a recognized word. (unrecognized-spelling)
smart_enable: all
- pymdownx.caret
- pymdownx.details
- pymdownx.highlight:
anchor_linenums: true

Check warning on line 134 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`linenums` is not a recognized word. (unrecognized-spelling)
line_spans: __span
pygments_lang_class: true
- pymdownx.inlinehilite
- pymdownx.keys
- pymdownx.magiclink:

Check warning on line 139 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`magiclink` is not a recognized word. (unrecognized-spelling)
normalize_issue_symbols: true
repo_url_shorthand: true
user: squidfunk

Check warning on line 142 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`squidfunk` is not a recognized word. (unrecognized-spelling)
repo: mkdocs-material
- pymdownx.mark
- pymdownx.smartsymbols
Expand All @@ -161,7 +161,7 @@
custom_checkbox: true
- pymdownx.tilde
- pymdownx.emoji:
emoji_index: !!python/name:materialx.emoji.twemoji

Check warning on line 164 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`materialx` is not a recognized word. (unrecognized-spelling)
emoji_generator: !!python/name:materialx.emoji.to_svg

# Plugins
Expand All @@ -180,7 +180,7 @@

nav:
- Home: index.md
- GenAI on Vertex AI:

Check warning on line 183 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`GenAI` matches a line_forbidden.patterns entry: `\b[gG]enAI\b`. (forbidden-pattern)
- Overview: genai-on-vertex-ai/README.md
- Foundation Models:
- Gemini 2.0:
Expand All @@ -207,7 +207,7 @@
- Advanced Prompting Training:
- Overview: genai-on-vertex-ai/advanced_prompting_training/README.md
- CoT & ReAct: genai-on-vertex-ai/advanced_prompting_training/cot_react.ipynb
- Vertex Foundation Tuning:

Check warning on line 210 in docs/mkdocs.yml

View workflow job for this annotation

GitHub Actions / Check Spelling 

`Vertex` matches a line_forbidden.patterns entry: `\b(?!Vertex AI\b)(?!.*[\(\)\{\},])(?<!import\s)(?<!\.)(?<!,\s)Vertex\s?[Aa]?[Ii]?\b`. (forbidden-pattern)
- Overview: genai-on-vertex-ai/vertex_foundation_tuning/README.md
- Model Tuning: genai-on-vertex-ai/vertex_foundation_tuning/vertexai-model-tuning.ipynb
- Evaluation:
Expand Down Expand Up @@ -249,6 +249,7 @@
- Inline Ingestion of Documents: genai-on-vertex-ai/vertex_ai_search/inline_ingestion_of_documents.ipynb
- Parsing and Chunking: genai-on-vertex-ai/vertex_ai_search/parsing_and_chunking_with_BYO.ipynb
- Query Level Boosting: genai-on-vertex-ai/vertex_ai_search/query_level_boosting_filtering_and_facets.ipynb
- Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores: genai-on-vertex-ai/vertex_ai_search/manual_recrawl_urls_with_trigger.ipynb
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

For consistency with other entries, consider capitalizing "Event-based" and "Trigger". Also, consider rephrasing to be more concise, perhaps "Triggering Manual Recrawls for Vertex AI Search (Advanced Website Datastores)"

Suggested change
- Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores: genai-on-vertex-ai/vertex_ai_search/manual_recrawl_urls_with_trigger.ipynb
- Triggering Manual Recrawls for Vertex AI Search (Advanced Website Datastores): genai-on-vertex-ai/vertex_ai_search/manual_recrawl_urls_with_trigger.ipynb

- DIY RAG:
- Overview: genai-on-vertex-ai/retrieval_augmented_generation/README.md
- Building Grounded RAG App: genai-on-vertex-ai/retrieval_augmented_generation/diy_rag_with_vertexai_apis/build_grounded_rag_app_with_vertex.ipynb
Expand Down
5 changes: 4 additions & 1 deletion genai-on-vertex-ai/vertex_ai_search/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@

# What is Vertex AI Search?

Vertex AI Search (VAIS) is a fully-managed platform, powered by large language models, that lets you build AI-enabled search and recommendation experiences for your public or private websites or mobile applications

VAIS can handle a diverse set of data sources including structured, unstructured, and website data, as well as data from third-party applications such as Jira, Salesforce, and Confluence.

VAIS also has built-in integration with LLMs which enables you to provide answers to complex questions, grounded in your data

# Sample Notebooks

This folder contains a series of notebooks to demonstrate how different functionalities within Vertex AI Search can be used

We aim to keep these notebooks broader than a single API call and smaller than a fully fledged application.
Expand All @@ -16,8 +17,10 @@ The notebooks are expected to serve as building blocks which can be combined to
We will try to use REST APIs which will hopefully make the codes easier to understand without a need to read through documentations of different object types. For production use, many customer prefer Client libraries. Please consult the [official documentation](https://cloud.google.com/generative-ai-app-builder/docs/apis) for alternative ways of achieving the same goals.

# List of Notebooks

1. [Ingestion of Unstructured Documents with Metadata in Vertex AI Search](./ingesting_unstructured_documents_with_metadata.ipynb)
2. [Parsing and Chunking in Vertex AI Search: Featuring BYO Capabilities](./parsing_and_chunking_with_BYO.ipynb)
3. [Defining custom attributes based on URL patterns in Vertex AI Search Website Datastores](./custom_attributes_by_url_pattern.ipynb)
4. [Query-Level Boosting, Filtering, and Facets for Vertex AI Search Website Datastores](./query_level_boosting_filtering_and_facets.ipynb)
5. [Inline Ingestion of Documents into Vertex AI Search](./inline_ingestion_of_documents.ipynb)
6. [Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores](./manual_recrawl_urls_with_trigger.ipynb)
405 changes: 405 additions & 0 deletions genai-on-vertex-ai/vertex_ai_search/manual_recrawl_urls_with_trigger.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

Loading