Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

End to end test doesn't exist #91

Open
visch opened this issue Jul 13, 2023 · 0 comments
Open

End to end test doesn't exist #91

visch opened this issue Jul 13, 2023 · 0 comments

Comments

@visch
Copy link
Member

visch commented Jul 13, 2023

I was trying to show that Windows jobs failed via Github actions while solving #87 , but the job succeeded when running on Windows. I know for a fact running on Windows and pushing data to snowflake doesn't work e2e right now without #89 merged

Maybe we could do a test like https://github.com/MeltanoLabs/target-postgres/blob/89b2b957d136492a2b71a176fef74e610f60b934/.github/workflows/ci_workflow.yml#L99C1-L100C54

Needed to modify meltano.yml to this to get a test going locally

version: 1
send_anonymous_usage_stats: true
project_id: target-snowflake
plugins:
  extractors:
  - name: tap-smoke-test
    variant: meltano
    pip_url: git+https://github.com/meltano/tap-smoke-test.git
    config:
      streams:
      - stream_name: animals
        input_filename: https://gitlab.com/meltano/tap-smoke-test/-/raw/main/demo-data/animals-data.jsonl
      - stream_name: pageviews
        input_filename: https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/pageviews-data.jsonl
  loaders:
  - name: target-snowflake
    namespace: target_snowflake
    pip_url: -e .
    capabilities:
    - about
    - schema-flattening
    - stream-maps
    settings_group_validation:
    - - account
      - database
      - password
      - user
    settings:
    - name: account
      kind: string
      label: Account
      description: Your account identifier. See [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html).
    - name: add_record_metadata
      kind: boolean
      value: true
      label: Add Record Metadata
      description: Whether to add metadata columns.
    - name: clean_up_batch_files
      kind: boolean
      value: true
      label: Clean Up Batch Files
      description: Whether to remove batch files after processing.
    - name: database
      kind: string
      label: Database
      description: The initial database for the Snowflake session.
    - name: default_target_schema
      kind: string
      label: Default Target Schema
      description: The default target database schema name to use for all streams.
    - name: flattening_enabled
      kind: boolean
      label: Flattening Enabled
      description: "'True' to enable schema flattening and automatically expand nested\
        \ properties."
    - name: flattening_max_depth
      kind: integer
      label: Flattening Max Depth
      description: The max depth to flatten schemas.
    - name: password
      kind: password
      label: Password
      description: The password for your Snowflake user.
    - name: role
      kind: string
      label: Role
      description: The initial role for the session.
    - name: schema
      kind: string
      label: Schema
      description: The initial schema for the Snowflake session.
    - name: stream_map_config
      kind: object
      label: Stream Map Config
      description: User-defined config values to be used within map expressions.
    - name: stream_maps
      kind: object
      label: Stream Maps
      description: Config object for stream maps capability. For more information
        check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html).
    - name: user
      kind: string
      label: User
      description: The login name for your Snowflake user.
    - name: warehouse
      kind: string
      label: Warehouse
      description: The initial warehouse for the session.
    config:
      start_date: '2010-01-01T00:00:00Z'
default_environment: dev
environments:
  - name: dev
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant