All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog.
- Updated 3W data for AFG, NER, TCD
- Updated population pipeline to read from global dataset
- Updated 3W data for BFA and COL
- Read poverty rate data from global dataset at national and subnational level
- Centralized error handling and added function to write errors to HDX resource metadata
- Updated 3W data for SSD
- Updated 3W data for MOZ, ETH, SDN, SSD
- Food security p-code mappings for ZAF
- Second Haiti population update
- Date issue in IDPs pipeline
- Haiti population update
- Errors in some food security data p-code mappings
- Added global food security data with additional p-code mappings
- Error in handling admin2_ref in humanitarian needs
- Use freeform category for humanitarian needs
- Populate provider_admin1_name and provider_admin2_name
- Output error for blank appeal code
- Lebanon operational presence
- Added check for null provider name
- Update subnational data to include provider admin 1 and 2 names
- Include global data for food prices, IDPs, conflict event, and food security
- Fix to use deprecated humanitarian needs resource temporarily
- Fix broken database export
- WFP commodity normalised and fuzzy matching
- Fix broken tests
- Melanie's food security changes (SOM admin 1 only)
- Fix how pipeline appears in MixPanel
- Split refugees into refugees and returnees
- IDP scraper
- Fix for TCD population dataset change (again)
- Info on IPC country processing
- Fix for TCD population dataset change
- Changes in HDX Python Scraper package names
- Updated Nigeria population and tests
- Fixed view names in export
- Updated El Salvador population resource names
- P-coded food security data from IPC
- Updated Guatemala population resource names
- 3W updates for Afghanistan, Cameroon, CAR, Ethiopia, Mali , Nigeria, South Sudan, Sudan, and Yemen
- Updated Colombia population resource names
- Remove Ukraine population resource which is now by request only
- Fix logging of HNO warnings and errors
- Read HNO data from global annual dataset with multiple resources
- Updated HNO file name pattern
- Read HNO data from global annual dataset(s)
- Updated BFA operational presence data
- Updated SDN population data
- Reenable fuzzy matching
- Convert xlsx to csv for conflict events
- Split pipelines test into multiple tests
- COD and TCD operational presence update
- Honduras operational presence file update
- Honduras operational presence file update
- Refactor org code
- Also add uncleaned names as keys to lookups
- Niger operational presence file update
- Normalise keys in sector and org type lookups on creation
- Fold unofficial mappings into self.data in sector and org type
- Simplify get_code_from_name
- Add 'has_hrp' and 'in_gho' fields to location table
- HND population resource names have been updated
- Update HDX Python Scraper to 2.4.0 - it has a small optimisation to the filtering and sorting of rows in the RowParser and a larger one which defaults fill_merged_cells to False so that Frictionless uses OpenPyXL in read only mode
- Update filenames for ETH conflict data
- Corrected changed Mozambique operational presence parameters
- Corrected changed Mozambique operational presence parameters
- Update Mozambique operational presence data to most recent
- Add missing sector mappings in operational presence data
- Update Ethiopia operational presence data to most recent
- get_pcode normalisation improvements
- Update filenames for ETH conflict data
- Use normalise function from HDX Python Utilities
- Update mappings for changes in HDX Python Country
- Use latest operational presence data for NER and SDN
- Added parameter to make phonetic matching optional for org types and sectors
- Added clean_text function to utilities
- Removed outdated error messages from operational presence pipeline
- Add check for funding requirements value due to missing data for UKR
- Updated Sudan 3W filename
- Specify countries per theme when running via the command line
- Thanks to bugfix in
openpyxl, added 3W data for: BFA, CAF, COD, NER - Updated 3W data for: MLI, SOM, ETH, SDN, SSD
- Avoid overwriting orgs with multiple org types
- Replace "POP" with "all" for humanitarian needs population status
- Update to latest schema version
- Switch HNO to use prod HDX data
- Non-standard age categories to population data
- Admin mappings for food prices data
- Removed filter on refugee data
- Added missing population data for SLV
- Function to add data to tables in batches
- Updated operational presence data for AFG
- Increased speed of conflict event pipeline and included data for all HRP countries
- Use schema with fixed enum mappings
- Roll ups use 'all' instead of '*'
- 'unknown' age range
- Replaced line by line commit with batch commit for operational presence and org pipelines
- Missing poverty rate countries: HND, SLV, GTM
- Currencies
- Commodities
- Markets
- Food prices
- Temporarily reduce data volume in conflict pipeline
- Filter miscoded unit in conflict data
- Correct overwriting issue in conflict pipeline
- Conflict event theme for all HRP countries
- Poverty rate theme for all countries where Oxford MPI is available
- Ability to specify more default parameters in yaml
- Output operational presence errors at end
- Logging functions for consistent output format
- HNO data and tests
- Funding data and tests
- Refugees data and tests
- HXL tag parsing functions to get age and gender
- Extended locations to global coverage
- Don't do phonetic matching for names of 5 characters or less
- Many changes to align with V1 of the schema
- ipc_phase, ipc_type, age_range, gender classes
- Use hdx id as primary key for resource and dataset tables
- Added hapi_updated_date fields to relevant tables
- Updated test data for humanitarian needs theme
- Updated operational presence data for Colombia
- Output views
- HAPI patch utility
- HAPI patch utility
- Updated operational presence theme to better match organizations
- Added all HRP countries to operational presence theme
- Added national_risk_view, humanitarian_needs_view, population_group_view, population_status_view
- Added all HRP countries to food security theme
- Added all HRP countries to national risk theme
- Linked national risk to admin 2 level
- Set countries to run for each theme for testing
- Allow dates to be specified in scraper config
- Add population data for all HRP countries
- Change date in org table to match v1 release date
- Correct outdated admin logic in operational presence
- Add national risk AFG, BFA, MLI, NGA, TCD, YEM
- Fix db export (wrong codes being used for age range)
- Fix for humanitarian needs TCD
- Add humanitarian needs AFG, TCD, YEM
- Use better pcode length conversion from HDX Python Country
- Add food security NGA
- When phase population is 0, set population_fraction_in_phase to 0.0
- Org mapping table to deduplicate orgs
- Fuzzy matching for sector and org types
- Limit AdminLevel countries
- Minor unit tests
- Food security and related tables for Burkina Faso, Chad, and Mali
- Resource filename changed to name
- HDX provider code and name change
- DB export GitHub action pushes to branch db-export
- Rename resource "filename" to "name" in metadata
- Default fields for configurations files
- Build views in pipeline instead of in hapi-schema
- DB Export GitHub action runs on tag push
- Add operational presence code matching to funtion in utilities
- Pinned postgres docker image version in DB export GitHub action
- Change sector mapping for erl to ERY
- Update requirements to use latest
hapi-schema - Change DB export GitHub action to have the HDX API key, and
to run
pg_dumpin the postgres docker container
- GitHub Action to create DB export
- Sector and org_type mappings
- Remove duplicates from operational presence
- Org type module name from schemas library
- Remove HDX link from org
- Operational presence resource ref
- Splitting of configs files
- 3W data ingestion