Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2.0.0 #1732

Open
wants to merge 233 commits into
base: master
Choose a base branch
from
Open

V2.0.0 #1732

Changes from all commits
Commits
Show all changes
233 commits
Select commit Hold shift + click to select a range
77540ab
[infra] Version 1.7.0 python-package
lucascr91 Nov 3, 2022
b0f5dfe
[infra] fix update_columns test
lucascr91 Nov 3, 2022
17ba1ba
[infra] remove unused import
lucascr91 Nov 3, 2022
a3f71df
[infra] add to_partition utility function
lucascr91 Nov 7, 2022
91ea441
[infra] add test for to_partitions
lucascr91 Nov 7, 2022
c51b4aa
Merge branch 'master' of https://github.com/basedosdados/mais into v1…
lucascr91 Nov 8, 2022
0cba449
[infra] pump package version 1.6.9-b2
lucascr91 Nov 8, 2022
6aaa1b1
[infra] add break_file feature
lucascr91 Nov 9, 2022
bef2953
Revert "[infra] pump package version 1.6.9-b2"
mfagundes Nov 9, 2022
2640684
feat: add `connection_id` to external data configuration
gabriel-milan Nov 30, 2022
dffb8ab
fix(Datatype): add connection id for external configuration
gabriel-milan Nov 30, 2022
cbfdeca
feat: add automatic management of BQ connection
gabriel-milan Nov 30, 2022
d10cee8
chore: fix linting issues
gabriel-milan Nov 30, 2022
69555f0
feat: add test folder to gitignore
d116626 Dec 1, 2022
4ca854f
feat: release beta version
d116626 Dec 1, 2022
61a6f88
feat(Connection): add `service_account` property
gabriel-milan Dec 1, 2022
34fe6eb
feat(Base): add IAM stuff
gabriel-milan Dec 1, 2022
5122127
chore: fix linting issues
gabriel-milan Dec 1, 2022
e701d06
feat: automatic granting roles to BigLake service account
gabriel-milan Dec 1, 2022
43f7913
feat: better error handling, set biglake permissions is now optional
gabriel-milan Dec 1, 2022
39e700a
feat: release beta version
d116626 Dec 1, 2022
8294db7
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Dec 5, 2022
1f4ab2e
chore: modify log message
gabriel-milan Dec 6, 2022
99c0911
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Dec 7, 2022
69ed7ef
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Dec 8, 2022
7a2536e
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Dec 12, 2022
d2d6b28
chore: make all partitions string
d116626 Dec 12, 2022
e468f5d
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Dec 13, 2022
2f61e97
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Dec 14, 2022
278cbda
Merge branch 'master' into v1.7.0
d116626 Dec 14, 2022
77bd36f
chore: merge master
d116626 Dec 14, 2022
5b74d01
Merge branch 'v1.7.0' into feat/add-biglake-tables
d116626 Dec 14, 2022
417aa6e
add __version__ atribute (#1488)
fernandascovino Dec 14, 2022
0e74206
add option to change copied table name (#1489)
fernandascovino Dec 14, 2022
a9dccb9
Merge branch 'v1.7.0' into feat/add-biglake-tables
mergify[bot] Dec 14, 2022
1f651d6
fix: pylint
d116626 Dec 14, 2022
a904c66
Merge branch 'v1.7.0' into feat/add-biglake-tables
mergify[bot] Dec 14, 2022
70acbda
changing python_path fixture
mfagundes Dec 21, 2022
2da91f8
adding shapely as dependency to downgrade if already installed, like …
mfagundes Dec 21, 2022
78c3574
pylinting files
mfagundes Dec 21, 2022
cad1400
Merge branch 'master' into fix/shapely-dependency
mergify[bot] Dec 22, 2022
fc983fa
bump version 1.6.10-beta.1
mfagundes Dec 22, 2022
221a137
Merge remote-tracking branch 'origin/fix/shapely-dependency' into fix…
mfagundes Dec 22, 2022
a570f45
Merge branch 'master' into fix/shapely-dependency
mfagundes Jan 9, 2023
4e6792b
bump version 1.6.10
mfagundes Jan 9, 2023
3c16959
Merge branch 'master' into python-1.6.10
mfagundes Jan 12, 2023
0e530da
Merge branch 'master' into v1.7.0
mfagundes Jan 17, 2023
667fce1
updating version
mfagundes Jan 17, 2023
8b2e5e6
Merge branch 'python-1.6.10' into v1.7.0
mfagundes Jan 23, 2023
4e4f381
return update_columns to Table class
mfagundes Jan 23, 2023
2cab54d
authentication methods in base class
mfagundes Jan 25, 2023
c427511
method to return dataset id from slug using graphql
mfagundes Jan 25, 2023
bc038d4
method to return table id from slug of dataset and table using graphql
mfagundes Jan 25, 2023
218aea9
using variables in graphql query
mfagundes Jan 25, 2023
119c7b6
change default & log downloaded path
fernandascovino Jan 26, 2023
ed55994
Merge branch 'master' into fix/python-storage-download-path
mergify[bot] Jan 26, 2023
d44c874
authentication with graphql
mfagundes Jan 26, 2023
cca4333
change version in pyproject.toml
mfagundes Jan 27, 2023
7349e93
fix: merge master
d116626 Jan 27, 2023
dbf3b91
chore: refactor connection imports and make working dir default for s…
d116626 Jan 27, 2023
04216fe
chore: make staging the default mode for storage download
d116626 Jan 27, 2023
5c337cd
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Jan 27, 2023
0ee6dad
chore: make staging the default mode for storage download
d116626 Jan 27, 2023
3b286a4
Merge branch 'feat/add-biglake-tables' of https://github.com/basedosd…
d116626 Jan 27, 2023
b8f7b6e
fix: pylint
d116626 Jan 27, 2023
7bb5f16
chore: release new beta version
d116626 Jan 27, 2023
1a956f0
small corrections in 1.6.11
mfagundes Jan 28, 2023
06c28cb
Merge branch 'master' into python-1.6.11
mfagundes Jan 28, 2023
37f7d6a
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Jan 30, 2023
07ee69a
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Jan 30, 2023
6726d6d
methods to retrieve metadata from graphql api
mfagundes Jan 31, 2023
17345c5
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Jan 31, 2023
6987083
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Jan 31, 2023
6569ac1
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Jan 31, 2023
f14d882
adding data to api_data_dict
mfagundes Feb 1, 2023
b297d24
adding exists_in_api method
mfagundes Feb 1, 2023
b72f65a
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 1, 2023
6e1d35d
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 1, 2023
bcc147c
changing is_updated method
mfagundes Feb 1, 2023
8155b76
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 1, 2023
7930e25
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 2, 2023
b613043
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 2, 2023
aee249a
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 2, 2023
fa7f28f
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 6, 2023
7f6686c
method to get a request in graphql
mfagundes Feb 6, 2023
cbe8acd
logging errors with loguru, instead of print
mfagundes Feb 6, 2023
83e44f0
writing yaml files before updating the database
mfagundes Feb 8, 2023
b075602
helper to convert case from snake_case to camelCase and vice versa
mfagundes Feb 8, 2023
2600f47
start refactoring the query to use alias
mfagundes Feb 9, 2023
dd6ef1a
refactoring: clean edges and nodes from graphql response
mfagundes Feb 9, 2023
4bb261d
moving graphql queries for separated files and others
mfagundes Feb 12, 2023
dcfd441
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 13, 2023
2b2f1b1
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 13, 2023
e57db4c
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 14, 2023
3d5ebe9
improving unit tests and graphql for api_metadata
mfagundes Feb 14, 2023
92ca3d4
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 14, 2023
4d6c364
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 15, 2023
dc97bd6
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 16, 2023
ad0f30f
changing api_response for compatibility with current yamls
mfagundes Feb 17, 2023
0e6a386
Hotfix storage init args (#1576)
mfagundes Feb 17, 2023
76dc755
Merging version 1.6.11 into 1.7.0
mfagundes Feb 17, 2023
ca47458
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 22, 2023
83c452d
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 24, 2023
c993639
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Feb 27, 2023
8ff247b
removing coverage from query
mfagundes Feb 28, 2023
e029536
fixes in dataset and table config files creation
mfagundes Feb 28, 2023
df57a7a
owner_org and exists_in_api methods
mfagundes Feb 28, 2023
aa14e0e
removing references to REST API and treat errors in login
mfagundes Mar 1, 2023
7d98b9e
adjustments in data_dict, tests for publish
mfagundes Mar 1, 2023
92464a8
initializing RemoteAPI for mutations
mfagundes Mar 1, 2023
6d7cba6
mutation to create a dataset
mfagundes Mar 2, 2023
7833b29
adjustments in data_dict and others
mfagundes Mar 3, 2023
3d2928d
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 4, 2023
7ac8b9b
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 6, 2023
2baf9d1
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 6, 2023
a3d798b
corrections and new table for publish tests
mfagundes Mar 7, 2023
42cea57
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 7, 2023
b3ecbb6
Merge branch 'master' into v1.7.0
mfagundes Mar 8, 2023
5a8fef9
removing IDE settings from project
mfagundes Mar 8, 2023
6c2a003
prevent exclusion of tmp_bases when it already exists
mfagundes Mar 8, 2023
d7af38b
change filename to avoid conflict in tests
mfagundes Mar 8, 2023
bc59197
commiting notebook file as it does not rollback, despite the fact tha…
mfagundes Mar 8, 2023
0960516
stable exists method
mfagundes Mar 9, 2023
fb3aaa4
metadata_modified as datetime
mfagundes Mar 10, 2023
ca101cc
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 13, 2023
996d732
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 14, 2023
6499f23
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 15, 2023
9245b43
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 15, 2023
ca4a61e
Merge branch 'master' into feat/add-biglake-tables
mergify[bot] Mar 15, 2023
9f6d686
chore: remove yaml dependecy from metadata.py
d116626 Apr 3, 2023
fa7c081
chore: remove yaml dependecy from metadata.py
d116626 Apr 3, 2023
507621b
feat: remove more code
d116626 Apr 3, 2023
a216095
feat: remove more code
d116626 Apr 3, 2023
be52f98
initial structure
d116626 Apr 5, 2023
4588881
chore: refactor graphql requests
d116626 Apr 5, 2023
d04d88b
chore: refactor graphql requests
d116626 Apr 5, 2023
107d65d
chore: refactor graphql requests
d116626 Apr 5, 2023
ad33499
chore: make publish_sql
d116626 Apr 10, 2023
a471546
feat: add backend class for handling interaction with graphql
gabriel-milan May 8, 2023
58b091b
chore: clean some code and comment parts where table_config are needed
d116626 May 9, 2023
e78e8ec
chore: clean some code and comment parts where table_config are needed
d116626 May 9, 2023
c711bd7
chore: add dataset config query
d116626 May 9, 2023
723436b
chore: add table config query
d116626 May 9, 2023
325efc7
feat: create dataset and use API metadata
d116626 May 9, 2023
1327588
chore(deps): remove unnecessary deps
gabriel-milan May 9, 2023
626a83f
chore: minor cleanup
gabriel-milan May 9, 2023
79205e8
chore: delete file
gabriel-milan May 9, 2023
743491c
feat: fix occurences of `table_config`
gabriel-milan May 9, 2023
bd48905
feat: add API url to config init
gabriel-milan May 9, 2023
e7eb09f
feat: add structure for `Metadata.create`
gabriel-milan May 9, 2023
54902fb
chore: more table modifications
d116626 May 10, 2023
31c42fc
merge biglake branch
d116626 May 10, 2023
87b2c0b
feat: table create using data_columns and partitioned data
d116626 May 10, 2023
68aa341
feat: some refactor and finish table.create
d116626 May 10, 2023
c01d07a
chore: better casing
d116626 May 10, 2023
e18a528
chore: better casing
d116626 May 10, 2023
571c070
chore: better logging
d116626 May 10, 2023
89e7c9f
chore: update table.create docstring
d116626 May 10, 2023
ba81cfa
chore: clean config files
d116626 May 10, 2023
2e96ee5
feat: refactor table.publish and table.update
d116626 May 11, 2023
acf67a4
chore: make publish.sql from staging schema
d116626 May 11, 2023
87f85aa
feat: get partition dict from storage
d116626 May 11, 2023
0b68fdd
chore: rename some methods
d116626 May 11, 2023
4e753f2
chore: update and publish only acts in prod and uses the staging tabl…
d116626 May 11, 2023
3a974ea
chore: load schema using SchemaField, remove code that depends on tem…
d116626 May 12, 2023
c68b47e
chore: refactor init process
d116626 May 12, 2023
d324a25
chore: remove upload function from cli
d116626 May 12, 2023
510aaf9
chore: remove upload function from cli
d116626 May 12, 2023
b4f8f03
chore: clean unused imports, redo poetry packages and release 2.0.0-b1
d116626 May 12, 2023
4ab9959
chore: add a new dependencie requests-toolbelt
d116626 May 13, 2023
3f7f2ac
chore: add tomlkit and better error if columns does not have name
d116626 May 13, 2023
0ec5286
chore: error handling and make publish and update get info from api …
d116626 May 15, 2023
77d522e
fix: typo in _get_columns_from_data and better infos
d116626 May 15, 2023
82eeeef
chore: add tomlkit
d116626 May 15, 2023
76a1637
chore: error handling in case that API is off
d116626 May 15, 2023
0e97037
chore: error handling in case that API is off
d116626 May 16, 2023
6355196
hotfix: chang metadata base_url
d116626 May 17, 2023
09ecafc
chore: get backend metadata from cloud tables
d116626 May 17, 2023
144e08e
feat: bump beta version
d116626 May 17, 2023
b7e825b
chore: change mode in table.delete
d116626 May 19, 2023
47efb6c
chore: pump version
d116626 May 19, 2023
bf9320e
chore: no more version number on files
gabriel-milan Jun 26, 2023
acb5854
feat: implement external warnings and messages
gabriel-milan Jun 26, 2023
db51c49
feat: add csv_delimiter and allow csv_allow_jagged_rows
d116626 Aug 21, 2023
7c6ee34
fix: pump bd version and add new parameters csv_delimiter and csv_all…
d116626 Aug 21, 2023
6ccceda
chore: cleanup
gabriel-milan Aug 23, 2023
0863bdb
chore: remove compressed r package
gabriel-milan Aug 23, 2023
bb48ea2
chore: refactor dependency management
gabriel-milan Aug 24, 2023
39ad10d
chore: fix linting issues
gabriel-milan Aug 24, 2023
ce96f58
chore: remove pylint action
gabriel-milan Aug 24, 2023
3748388
fix: change install instructions
d116626 Aug 24, 2023
b98aeb6
feat: pump bd version
d116626 Aug 24, 2023
f3e6c69
fix: change install instructions
d116626 Aug 24, 2023
00dc835
feat: pump bd version
d116626 Aug 24, 2023
6b937ca
feat: add new parameter csv_skip_leading_rows and setup.py
d116626 Aug 24, 2023
216894d
feat: pump bd version
d116626 Aug 24, 2023
33d492f
chore: merge branch v1.7.0 and feat/add-biglake-tables
d116626 Aug 24, 2023
884ce26
chore: merge master and fix conflicts
d116626 Aug 25, 2023
a791652
Merge branch 'master' into feat/remove_yamls
mergify[bot] Aug 25, 2023
75aa2f4
chore: fix conflicts
d116626 Aug 25, 2023
c33a431
Merge branch 'feat/remove_yamls' of https://github.com/basedosdados/m…
d116626 Aug 25, 2023
ef46c4d
chore: add `all` extra
gabriel-milan Aug 25, 2023
7369617
chore: lint
gabriel-milan Aug 25, 2023
146afc2
Merge branch 'master' into feat/remove_yamls
mergify[bot] Aug 28, 2023
7dca3b6
feat: create branch v2.0.0
d116626 Aug 30, 2023
c69f84d
Merge branch 'master' into v2.0.0
mergify[bot] Sep 2, 2023
2e856e9
fix: add csv delimiter to schema
d116626 Sep 12, 2023
50cd8ec
Merge branch 'v2.0.0' of https://github.com/basedosdados/mais into v2…
d116626 Sep 12, 2023
125f28f
Merge branch 'master' into v2.0.0
mergify[bot] Sep 21, 2023
0635373
Merge branch 'master' into v2.0.0
mergify[bot] Sep 26, 2023
61d2655
feat: expand credential scope to drive and bq
d116626 Oct 3, 2023
bb26637
Merge branch 'master' into v2.0.0
mergify[bot] Jan 8, 2024
84b8b5c
Merge branch 'master' into v2.0.0
mergify[bot] Jan 11, 2024
ea62797
Merge branch 'master' into v2.0.0
mergify[bot] Jan 21, 2024
14c0ee9
chore: start cleaning tests
d116626 Feb 20, 2024
653f256
Merge branch 'master' into v2.0.0
mergify[bot] Mar 5, 2024
ed85bb5
Merge branch 'master' into v2.0.0
mergify[bot] Mar 23, 2024
dab0051
Merge branch 'master' into v2.0.0
mergify[bot] Apr 1, 2024
4562113
chore: add timeout to pypi warning
d116626 Apr 2, 2024
ce941af
fix: log in wrong position
d116626 Jul 24, 2024
f6b4265
fix: log in wrong position
d116626 Jul 24, 2024
1918719
fix: log in wrong position
d116626 Jul 24, 2024
aae3cf8
fix: fix conflicts
d116626 Jul 24, 2024
f27f4d3
fix: fix conflicts
d116626 Jul 24, 2024
c61cf96
Fix #1718 (#1719)
aspeddro Jul 24, 2024
0baab83
feat: pump bd version
d116626 Jul 24, 2024
b925445
Merge branch 'v2.0.0' of https://github.com/basedosdados/mais into v2…
d116626 Jul 24, 2024
7a91bc3
chore: remove deprecated actions
d116626 Jul 24, 2024
e998c82
chore: add core.base in init
d116626 Jul 24, 2024
42c86d5
chore: add core.base in init
d116626 Jul 24, 2024
abf73f2
chore: add core.base in init
d116626 Jul 24, 2024
4302b9d
Merge branch 'master' of https://github.com/basedosdados/mais into v2…
d116626 Jul 24, 2024
5e11cac
fix `_set_config_variables` is not defined (#1729)
aspeddro Sep 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions python-package/basedosdados/download/download.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""
Functions for managing downloads
"""

import gzip
import os
import re
Expand All @@ -26,6 +27,19 @@
from pydata_google_auth.exceptions import PyDataCredentialsError


def _set_config_variables(billing_project_id, from_file):
"""
Set billing_project_id and from_file variables
"""

# standard billing_project_id configuration
billing_project_id = billing_project_id or config.billing_project_id
# standard from_file configuration
from_file = from_file or config.from_file

return billing_project_id, from_file


def read_sql(
query,
billing_project_id=None,
Expand Down Expand Up @@ -67,9 +81,9 @@ def read_sql(

return read_gbq(
query,
project_id=config.billing_project_id,
project_id=billing_project_id,
use_bqstorage_api=use_bqstorage_api,
credentials=_credentials(from_file=config.from_file, reauth=reauth),
credentials=_credentials(from_file=from_file, reauth=reauth),
)
except GenericGBQException as e:
if "Reason: 403" in str(e):
Expand Down