Add streets table from data.gov

[atalyaalon]

The list in data.gov is here

1. Add it to our DB
2. Create a flow in Airflow that updates it once a week
3. Decide where to use it and how in our streets cache, discuss with @tkalir and me, and let's implement

[ziv17]

Hi @atalyaalon,

• this is still relevant, right?
• do you mean to download manually, or automatically as part of the insert?
• If automatically, is there an endpoint for API, or do I need to parse the HTML?
• If manually:
• Encoding: originally it is Hebrew(1255), and size 1.7M Converting to UTF-8, it takes 2.5M. So I assume we will store it in S3? Hebrew(1255) or UTF-8?

[atalyaalon]

@ziv17 I suggest you take this next task , since this is relevant for Tamar's work
(I know there is Streets table, not sure when and where it's used, but you can find this mistery and perhaps use this table for this load)

I suggest

• storing it directly in our DB (and no need to save the file itself in S3)
• use this file: https://data.gov.il/dataset/321/resource/9ad3862c-8391-4b2f-84a4-2d4c68625f4b - there is a button for "API לנתונים"
• Make sure it runs in Airflow once a month
• Regarding Encoding - how do we handle it today in our DB when loading to it hebrew strings from other flows?