Skip to content

Commit

Permalink
minor improvements, using real dataset
Browse files Browse the repository at this point in the history
  • Loading branch information
@sfc-gh-ralfaroviquez
sfc-gh-ralfaroviquez committed Jun 19, 2024
1 parent 2e9fc31 commit 60d421b
Show file tree
Hide file tree
Showing 6 changed files with 36 additions and 216 deletions.
97 changes: 5 additions & 92 deletions snowflake-cortex/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,95 +1,8 @@
## Introduction

This is an example template for a Snowflake Native App project which demonstrates the use of Python extension code and adding Streamlit code. This template is meant to guide developers towards a possible project structure on the basis of functionality, as well as to indicate the contents of some common and useful files.

Since this template contains Python files only, you do not need to perform any additional steps to build the source code. You can directly go to the next section. However, if there were any source code that needed to be built, you must manually perform the build steps here before proceeding to the next section.

Similarly, you can also use your own build steps for any other languages supported by Snowflake that you wish to write your code in. For more information on supported languages, visit [docs](https://docs.snowflake.com/en/developer-guide/stored-procedures-vs-udfs#label-sp-udf-languages).

## Run the app
Create or update an application package in your Snowflake account, upload application artifacts to a stage in the application package, and create or update an application object in the same account based on the uploaded artifacts.
```
snow app run
```

For more information, please refer to the Snowflake Documentation on installing and using Snowflake CLI to create a Snowflake Native App.
# Directory Structure
## `/app`
This directory holds your Snowflake Native App files.

### `/app/README.md`
Exposed to the account installing the application with details on what it does and how to use it.

### `/app/manifest.yml`
Defines properties required by the application package. Find more details at the [Manifest Documentation.](https://docs.snowflake.com/en/developer-guide/native-apps/creating-manifest)

### `/app/setup_script.sql`
Contains SQL statements that are run when a consumer installs or upgrades a Snowflake Native App in their account.

## `/scripts`
You can add any additional scripts such as `.sql` and `.jinja` files here. One common use case for such a script is to add shared content from external databases to your application package. This allows you to refer to the external database in the setup script that runs when a Snowflake Native App is installed.
_Note: As of now, `snow app init` does not render these jinja templates for you into the required files, if you decide to use them. You will have to manually render them for now._


## `/src`
This directory contains code organization by functionality, such as one distinct module for Streamlit related code, and another module for "number add" functionality, which is used an example in this template.
```
/src
|-module-add
| |-main
| | |-python
| | |-add.py
| |
| |-test
| |-python
| |-add_test.py
|
|-module-ui
| |-src
| |-ui.py
| |-environment.yml
| |-test
| |-test_ui.py
To run this example first execute this command:
```bash
snow sql -f 'prepare/prepare_data.sql'
```

## `snowflake.yml.jinja`
While this file exists as a Jinja template, it is the only file that is automatically rendered as a `snowflake.yml` file by the `snow app init` command, as described in the [README.md](../README.md). Snowflake CLI uses the `snowflake.yml` file to discover your project's code and interact with Snowflake using all relevant privileges and grants.

For more information, please refer to the Snowflake Documentation on installing and using Snowflake CLI to create a Snowflake Native App.

## Adding a snowflake.local.yml file
Although your project directory must have a `snowflake.yml` file, an individual developer can choose to customize the behavior of Snowflake CLI by providing local overrides to the `snowflake.yml` file, such as a new role to test out your own application package. This is where you can use the `snowflake.local.yml` file, which is not a version-controlled file.
Then run `snow app run` on your terminal.

For more information, please refer to the Snowflake Documentation on installing and using Snowflake CLI to create a Snowflake Native App.

## Unit tests
To set up and run unit tests, please follow the steps below.

### Set up testing conda environment (First Time setup)

Go to the project's root directory where you can find `local_test_env.yml` and run the following command once to set up a conda environment with the correct packages. Please note that the version of test packages may differ from the version of packages in Snowflake, so you will need to be careful with any differences in behavior.

```
conda env create --file local_test_env.yml
```

This will create a conda environment with the name `streamlit-python-testing`.

### Run unit tests
To run unit tests, follow these steps:

#### Activate conda environment
You will need to activate this conda environment once per command line session:
```
conda activate streamlit-python-testing
```
To deactivate and use your current command line session for other tasks, run the following:
```
conda deactivate
```
#### Run Pytest
To run the example tests provided, execute the following command from the project's root:
```
pytest
```
Note that there is a pytest.ini file specifying the location of the source code that we are testing.
Currently it has a very short dataset due to llm model bytes limit.
10 changes: 5 additions & 5 deletions snowflake-cortex/app/setup_script.sql
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ CREATE APPLICATION ROLE IF NOT EXISTS app_public;
-- 2. Create a versioned schema to hold those UDFs/Stored Procedures
CREATE OR ALTER VERSIONED SCHEMA core;
GRANT USAGE ON SCHEMA core TO APPLICATION ROLE app_public;
-- -- 3. Create a view referencing the shared view in the package shared-content.sql.
-- CREATE VIEW IF NOT EXISTS core.shared_view
-- AS SELECT * FROM package_shared.PROVIDER_SPOTIFY_VIEW;
-- 3. Create a view referencing the shared view in the package shared-content.sql.
CREATE VIEW IF NOT EXISTS core.shared_view
AS SELECT * FROM package_shared.PROVIDER_SPOTIFY_VIEW;

-- -- 4. Grant appropriate privileges over the view to your application roles.
-- GRANT SELECT ON VIEW core.shared_view TO APPLICATION ROLE app_public;
-- 4. Grant appropriate privileges over the view to your application roles.
GRANT SELECT ON VIEW core.shared_view TO APPLICATION ROLE app_public;

-- 3. Create a streamlit object using the code you wrote in you wrote in src/module-ui, as shown below.
-- The `from` value is derived from the stage path described in snowflake.yml
Expand Down
118 changes: 13 additions & 105 deletions snowflake-cortex/prepare/provider_data.sql
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
USE ROLE ACCOUNTADMIN;
CREATE DATABASE IF NOT EXISTS ;
CREATE DATABASE IF NOT EXISTS SPOTIFY_CORTEX_DB;
CREATE SCHEMA IF NOT EXISTS SPOTIFY_CORTEX_DB.SPOTIFY_CORTEX_SCHEMA;
CREATE OR REPLACE TABLE SPOTIFY_CORTEX_DB.SPOTIFY_CORTEX_SCHEMA.SPOTIFY_PROVIDER_DATA(
track_name VARCHAR,
Expand All @@ -9,109 +9,17 @@ CREATE OR REPLACE TABLE SPOTIFY_CORTEX_DB.SPOTIFY_CORTEX_SCHEMA.SPOTIFY_PROVIDER
released_month INTEGER,
released_day INTEGER,
in_spotify_charts INTEGER,
streams INTEGER,
danceability_% INTEGER
spotify_streams INTEGER,
danceability_percentage INTEGER
);
INSERT INTO SPOTIFY_CORTEX_DB.SPOTIFY_CORTEX_SCHEMA.SPOTIFY_PROVIDER_DATA VALUES
("Seven (feat. Latto) (Explicit Ver.)","Latto, Jung Kook",2,2023,7,14,147,141381703,80),
("LALA","Myke Towers",1,2023,3,23,48,133716286,71),
("vampire","Olivia Rodrigo",1,2023,6,30,113,140003974,51),
("Cruel Summer","Taylor Swift",1,2019,8,23,100,800840817,55),
("WHERE SHE GOES","Bad Bunny",1,2023,5,18,50,303236322,65),
("Sprinter","Dave, Central Cee",2,2023,6,1,91,183706234,92),
("Ella Baila Sola","Eslabon Armado, Peso Pluma",2,2023,3,16,50,725980112,67),
("Columbia","Quevedo",1,2023,7,7,43,58149378,67),
("fukumean","Gunna",1,2023,5,15,83,95217315,85),
("La Bebe - Remix","Peso Pluma, Yng Lvcas",2,2023,3,17,44,553634067,81),
("un x100to","Bad Bunny, Grupo Frontera",2,2023,4,17,40,505671438,57),
("Super Shy","NewJeans",1,2023,7,7,55,58255150,78),
("Flowers","Miley Cyrus",1,2023,1,12,115,1316855716,71),
("Daylight","David Kushner",1,2023,4,14,98,387570742,51),
("As It Was","Harry Styles",1,2022,3,31,130,2513188493,52),
("Kill Bill","SZA",1,2022,12,8,77,1163093654,64),
("Cupid - Twin Ver.","Fifty Fifty",1,2023,2,24,77,496795686,78),
("What Was I Made For? [From The Motion Picture 'Barbie']","Billie Eilish",1,2023,7,13,104,30546883,44),
("Classy 101","Feid, Young Miko",2,2023,3,31,40,335222234,86),
("Like Crazy","Jimin",1,2023,3,24,68,363369738,63),
("LADY GAGA","Gabito Ballesteros, Junior H, Peso Pluma",3,2023,6,22,26,86444842,65),
("I Can See You (Taylor\'s Version) (From The ","Taylor Swift",1,2023,7,7,38,52135248,69),
("I Wanna Be Yours","Arctic Monkeys",1,2013,1,1,110,1297026226,48),
("Peso Pluma: Bzrp Music Sessions, Vol. 55","Bizarrap, Peso Pluma",2,2023,5,31,40,200647221,85),
("Popular (with Playboi Carti & Madonna) - The Idol Vol. 1 (Music from the HBO Original Series)","The Weeknd, Madonna, Playboi Carti",3,2023,6,2,87,115364561,85),
("SABOR FRESA","Fuerza Regida",1,2023,6,22,26,78300654,79),
("Calm Down (with Selena Gomez)","Rema, Selena G",2,2022,3,25,77,899183384,80),
("MOJABI GHOST","Tainy, Bad Bunny",2,2023,6,29,40,61245289,81),
("Last Night","Morgan Wallen",1,2023,1,31,19,429829812,52),
("Dance The Night (From Barbie The Album)","Dua Lipa",1,2023,5,25,101,127408954,67),
("Rush","Troye Sivan",1,2023,7,13,78,22581161,74),
("TULUM","Peso Pluma, Grupo Frontera",2,2023,6,28,34,52294266,56),
("Creepin'","The Weeknd, 21 Savage, Metro Boomin",3,2022,12,2,88,843957510,71),
("Anti-Hero","Taylor Swift",1,2022,10,21,56,999748277,64),
("TQG","Karol G, Shakira",2,2023,2,23,49,618990393,72),
("Los del Espacio","Big One, Duki, Lit Killah, Maria Becerra, FMK, Rusherking, Emilia, Tiago pzk",8,2023,6,1,31,123122413,81),
("Fragil (feat. Grupo Front","Yahritza Y Su Esencia, Grupo Frontera",2,2023,4,7,34,188933502,61),
("Blank Space","Taylor Swift",1,2014,1,1,53,1355959075,75),
("Style","Taylor Swift",1,2014,1,1,42,786181836,60),
("TQM","Fuerza Regida",1,2023,5,19,28,176553476,79),
("El Azul","Junior H, Peso Pluma",2,2023,2,10,25,354495408,56),
("Sunflower - Spider-Man: Into the Spider-Verse","Post Malone, Swae Lee",2,2018,10,9,78,2808096550,76),
("I'm Good (Blue)","Bebe Rexha, David Guetta",2,2022,8,26,80,1109433169,56),
("See You Again","Tyler, The Creator, Kali Uchis",3,2017,7,21,64,1047101291,56),
("Barbie World (with Aqua) [From Barbie The Album]","Nicki Minaj, Aqua, Ice Spice",3,2023,6,23,80,65156199,77),
("Angels Like You","Miley Cyrus",1,2020,11,27,19,570515054,67),
("I Ain't Worried","OneRepublic",1,2022,5,13,76,1085685420,71),
("Die For You","The Weeknd",1,2016,11,24,59,1647990401,59),
("Starboy","The Weeknd, Daft Punk",2,2016,9,21,79,2565529693,68),
("Die For You - Remix","Ariana Grande, The Weeknd",2,2023,2,24,47,518745108,53),
("El Cielo","Feid, Myke Towers, Sky Rompiendo",3,2023,6,2,38,107753850,72),
("Baby Don't Hurt Me","David Guetta, Anne-Marie, Coi Leray",3,2023,4,6,66,177740666,60),
("AMARGURA","Karol G",1,2023,2,24,39,153372011,92),
("(It Goes Like) Nanana - Edit","Peggy Gou",1,2023,6,15,59,57876440,67),
("Another Love","Tom Odell",1,2012,10,15,83,1813673666,45),
("Blinding Lights","The Weeknd",1,2019,11,29,69,3703895074,50),
("Moonlight","Kali Uchis",1,2023,2,24,42,256483385,64),
("La Bachata","Manuel Turizo",1,2022,5,26,45,1214083358,84),
("S91","Karol G",1,2023,7,14,41,16011326,86),
("cardigan","Taylor Swift",1,2020,7,24,29,812019557,61),
("Ta OK","dennis, MC Kevin o Chris",2,2023,5,4,15,111947664,86),
("Boy's a liar Pt. 2","PinkPantheress, Ice Spice",2,2023,2,3,41,156338624,70),
("Left and Right (Feat. Jung Kook of BTS)","Charlie Puth, BTS, Jung Kook",3,2022,6,24,39,720434240,88),
("BESO","Rauw Alejandro, ROSAL�",2,2023,3,24,50,357925728,77),
("Hey Mor","Ozuna, Feid",2,2022,10,6,38,674072710,90),
("Yellow","Chris Molitor",1,1999,1,1,43,1755214421,43),
("Karma","Taylor Swift",1,2022,10,21,23,404562836,64),
("People","Libianca",1,2022,12,2,56,373199958,59),
("Overdrive","Post Malone",1,2023,7,14,36,14780425,56),
("Enchanted (Taylor's Version)","Taylor Swift",1,2023,7,7,24,39578178,51),
("BABY HELLO","Rauw Alejandro, Bizarrap",2,2023,6,23,35,54266102,77),
("Heat Waves","Glass Animals",1,2020,6,28,63,2557975762,76),
("golden hour","JVKE",1,2022,7,15,36,751134527,51),
("Sweater Weather","The Neighbourhood",1,2012,5,14,61,2282771485,61),
("Quevedo: Bzrp Music Sessions, Vol. 52","Bizarrap, Quevedo",2,2022,7,6,45,1356565093,62),
("Viva La Vida","Coldplay",1,2008,1,1,62,1592909789,49),
("Here With Me","d4vd",1,2022,7,17,23,635412045,58),
("Unholy (feat. Kim Petras)","Sam Smith, Kim Petras",2,2022,9,22,42,1230675890,71),
("Yandel 150","Yandel, Feid",2,2022,12,20,38,585695368,78),
("CORAZON VACIO","Maria Becerra",1,2023,6,22,20,43857627,68),
("Riptide","Vance Joy",1,1975,1,1,55,2009094673,48),
("Until I Found You (with Em Beihold) - Em Beihold Version","Em Beihold, Stephen Sanchez",2,2022,4,22,30,600976848,34),
("Novidade na area","Mc Livinho, DJ Matt D",2,2023,6,23,9,39709092,63),
("Back To December (Taylor's Version)","Taylor Swift",1,2023,7,7,17,39228929,50),
("STAY (with Justin Bieber)","Justin Bieber, The Kid Laroi",2,2021,7,9,36,2665343922,59),
("El Merengue","Marshmello, Manuel Turizo",2,2023,3,3,44,223633238,78),
("Someone You Loved","Lewis Capaldi",1,2018,11,8,53,2887241814,50),
("Me Porto Bonito","Chencho Corleone, Bad Bunny",2,2022,5,6,43,1440757818,91),
("Makeba","Jain",1,2015,6,22,53,165484133,82),
("MONTAGEM - FR PUNK","Ayparia, unxbected",2,2012,6,20,50,58054811,63),
("Fast Car","Luke Combs",1,2023,3,24,12,157058870,71),
("What It Is (Solo Version)","Doechii",1,2023,3,17,25,95131998,74),
("Coco Chanel","Bad Bunny, Eladio Carrion",2,2023,3,17,38,250305248,68),
("Don't blame me","Taylor Swift",1,2017,11,8,23,685032533,62),
("Still With You","Jung Kook",1,2020,6,5,39,38411956,53),
("All My Life (feat. J. Cole)","J. Cole, Lil Durk",2,2023,5,12,23,144565150,83),
("Say Yes To Heaven","Lana Del Rey",1,2023,3,17,46,127567540,49),
("Snooze","SZA",1,2022,12,9,25,399686758,56),
("Summertime Sadness","Lana Del Rey",1,2011,1,1,52,983637508,56),
("Take Two","BTS",1,2023,6,9,47,118482347,62),
("Lover","Taylor Swift",1,2012,1,1,23,882831184,43);

('Seven \(feat. Latto\) \(Explicit Ver.\)','Latto, Jung Kook',2,2023,7,14,147,141381703,80),
('LALA','Myke Towers',1,2023,3,23,48,133716286,71),
('vampire','Olivia Rodrigo',1,2023,6,30,113,140003974,51),
('Cruel Summer','Taylor Swift',1,2019,8,23,100,800840817,55),
('WHERE SHE GOES','Bad Bunny',1,2023,5,18,50,303236322,65),
('Sprinter','Dave, Central Cee',2,2023,6,1,91,183706234,92),
('Ella Baila Sola','Eslabon Armado, Peso Pluma',2,2023,3,16,50,725980112,67),
('Columbia','Quevedo',1,2023,7,7,43,58149378,67),
('fukumean','Gunna',1,2023,5,15,83,95217315,85),
('La Bebe - Remix','Peso Pluma, Yng Lvcas',2,2023,3,17,44,553634067,81);
6 changes: 3 additions & 3 deletions snowflake-cortex/scripts/shared_content.sql
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
-- Grant usage to a external database from the application.
grant reference_usage on database SPOTIFY_CORTEX_DB
to share in application package {{ package_name }};

create schema if not exists {{ package_name }}.PACKAGE_SHARED;
use schema {{ package_name }}.PACKAGE_SHARED;

grant reference_usage on database SPOTIFY_CORTEX_DB
to share in application package {{ package_name }};

-- Create a view that references the provider table.
-- The view is going to be shared by the package to the application.
create view if not exists PACKAGE_SHARED.PROVIDER_SPOTIFY_VIEW
Expand Down
6 changes: 3 additions & 3 deletions snowflake-cortex/snowflake.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ native_app:
dest: ./
- src: src/ui/*
dest: streamlit/
# package:
# scripts:
# - scripts/shared_content.sql
package:
scripts:
- scripts/shared_content.sql
15 changes: 7 additions & 8 deletions snowflake-cortex/src/ui/ui.py
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
import snowflake.snowpark.functions as F
import pandas as pd

def call_cortex(df, input: str):
def call_cortex(session, df, input: str):
sample_df = df.to_pandas().to_string()
input_with_table = f"""The following data is a table that contains songs information it is wrapped in this tags <dataset>{sample_df}</dataset> {input}"""
new_df = df.with_column("CORTEX", F.call_builtin("SNOWFLAKE.CORTEX.COMPLETE", F.lit('llama2-70b-chat'), F.lit(input_with_table))).collect()
return new_df
new_df = session.create_dataframe([[1]],schema=["A"])
new_df = new_df.with_column("CORTEX", F.call_builtin("SNOWFLAKE.CORTEX.COMPLETE", F.lit('llama2-70b-chat'), F.lit(input_with_table)))
return new_df.select(F.col("CORTEX"))

def run_streamlit():
# Import python packages
# Streamlit app testing framework requires imports to reside here
import pandas as pd
import streamlit as st
from snowflake.snowpark import Session

Expand All @@ -18,13 +19,11 @@ def run_streamlit():
# Get the current credentials
session = Session.builder.getOrCreate()

df1 = session.create_dataframe([[1, 2, 3, 4, 5]], schema=["number1","number2","number3","number4","number5"])
df1.write.save_as_table("my_table", mode="overwrite")
table_df = session.table("my_table")
table_df = session.table("core.shared_view")
st.write(table_df)
cortex_input = st.text_input("What do you want to ask to cortex about the table from above?")
if cortex_input is not "":
cortex_fn = call_cortex(table_df, cortex_input)
cortex_fn = call_cortex(session, table_df, cortex_input)
st.write(cortex_fn)

if __name__ == '__main__':
Expand Down

0 comments on commit 60d421b

Please sign in to comment.