Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Azerbaijani support #212

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

saymantech-co
Copy link

@saymantech-co saymantech-co commented Oct 18, 2021

Implement Azerbaijani support(az_AZ)

Type of PR

  • Feature implementation

‌Testing

  • pytest test/test_format_az.py
  • pytest test/test_parse_az.py

@devs-mycroft devs-mycroft added the CLA: Needed Need signed CLA from https://mycroft.ai/cla label Oct 18, 2021
@devs-mycroft
Copy link
Collaborator

Hello, @saymantech-co, thank you for helping with the Mycroft project! We welcome everyone
into the community and greatly appreciate your help as we work to build an AI
for Everyone.

To protect yourself, the project, and users of Mycroft technologies we require
a Contributor Licensing Agreement (CLA) before accepting any code
contribution. This agreement makes it crystal clear that along with your
code you are offering a license to use it within the confines of this project.
You retain ownership of the code, this is just a license.

Please visit https://mycroft.ai/cla to initiate this one-time signing. Thank
you!

@saymantech-co
Copy link
Author

Hello, @saymantech-co, thank you for helping with the Mycroft project! We welcome everyone into the community and greatly appreciate your help as we work to build an AI for Everyone.

To protect yourself, the project, and users of Mycroft technologies we require a Contributor Licensing Agreement (CLA) before accepting any code contribution. This agreement makes it crystal clear that along with your code you are offering a license to use it within the confines of this project. You retain ownership of the code, this is just a license.

Please visit https://mycroft.ai/cla to initiate this one-time signing. Thank you!

Signed the cla.

@devs-mycroft devs-mycroft added CLA: Yes Contributor License Agreement exists (see https://github.com/MycroftAI/contributors) and removed CLA: Needed Need signed CLA from https://mycroft.ai/cla labels Oct 19, 2021
@siathon
Copy link

siathon commented Nov 5, 2021

@krisgesling For the date time stuff to work, the "az_AZ" locale needs to be generated in the OS, that's why the tests failed. and it can't be done from inside the code, it requires root access. Is this acceptable or should I find another way to handle the date time stuff?

@krisgesling
Copy link
Contributor

Sorry I missed the notification for your last message.

My first reaction is that we should expect the distribution packaging this to ensure the correct locales are available, and yes we can add that to the CI job here too.

I'm not at all familiar with the locale module so not sure what side effects we might be creating by switching the locale back and forth. There are also warnings of it not being threadsafe.

Would it be sufficient for us to install language-pack-az?

@krisgesling
Copy link
Contributor

#216 looks like it generated the locale without issue, so we can try it but I'm very interested to hear from anyone who's worked with locale before. Or if there are other options we can go with.

Copy link
Contributor

@krisgesling krisgesling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey sorry it's taken a while to get to this.

It's always a challenge reviewing extensive code in another language! But it looks pretty great.

There are a few little questions below but nothing of concern. The only thing is whether we really need to set the locale or not.

lingua_franca/lang/common_data_az.py Show resolved Hide resolved
lingua_franca/res/text/az-az/date_time.json Show resolved Hide resolved
Comment on lines 221 to 230
# value is platform dependent so better not use in tests?
# self.assertEqual(
# pronounce_number(sys.float_info.min), "iki nöqtə iki iki times "
# "on to the power of "
# "mənfi üç yüz "
# "və səkkiz")
# self.assertEqual(
# pronounce_number(sys.float_info.max), "bir nöqtə yeddi doqquz "
# "times on to the power of"
# " üç yüz və səkkiz")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd agree with the comment - this can be deleted unless you see specific value in it?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, most of these comments are the result of using the english files as template. I'll review and remove them.

self.assertEqual(
pronounce_number(1000000000000, short_scale=True),
"bir trilyon")
# TODO maybe beautify this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this relevant to the AZ test or just cut and pasted from the other file?

Copy link

@siathon siathon Nov 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, sorry I used english files as template and edited them, and seems i have forgotten to remove some of non relevant comments. I'll review and remove them.

test/test_format_az.py Outdated Show resolved Hide resolved
@@ -0,0 +1,429 @@
#
# Copyright 2017 Mycroft AI Inc.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to use the current year in each of these license headers.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I will fix them.

@krisgesling krisgesling mentioned this pull request Nov 25, 2021
2 tasks
@siathon
Copy link

siathon commented Nov 25, 2021

Sorry I missed the notification for your last message.

My first reaction is that we should expect the distribution packaging this to ensure the correct locales are available, and yes we can add that to the CI job here too.

I'm not at all familiar with the locale module so not sure what side effects we might be creating by switching the locale back and forth. There are also warnings of it not being threadsafe.

Would it be sufficient for us to install language-pack-az?

The reason for switching the locale back and forth was when running pytest, after running AZ tests, because of changing the locale in AZ tests, all other language tests failed, So in AZ, I returned the locale to the previous locale.

if that is not necessary and only passing test_parse_az and test_format_az is enough, the switching back and forth is not required and can be removed.

@krisgesling
Copy link
Contributor

With the AZ language pack installed and all the locale setting removed from parse_az.py. The following 4 test cases fail:

        testExtract("3 avqustda anama zəng etməyi xatırlat",
                    "2017-08-03 00:00:00", "anama zəng etməyi xatırlat")
        testExtract("iyulun 4 də atəşfəşanlıq al",
                    "2017-07-04 00:00:00", "atəşfəşanlıq al")
        testExtract("dekabr 3",
                    "2017-12-03 00:00:00", "")
        testExtract("5 iyun 2017 ci il axşamı anama zəng etməyi xatırlat",
                    "2017-06-05 19:00:00", "anama zəng etməyi xatırlat")

Is that what you're seeing also?

Are these the only test cases that would rely on the locale getting set or does it point to something else going on?

@siathon
Copy link

siathon commented Nov 30, 2021

With the AZ language pack installed and all the locale setting removed from parse_az.py. The following 4 test cases fail:

        testExtract("3 avqustda anama zəng etməyi xatırlat",
                    "2017-08-03 00:00:00", "anama zəng etməyi xatırlat")
        testExtract("iyulun 4 də atəşfəşanlıq al",
                    "2017-07-04 00:00:00", "atəşfəşanlıq al")
        testExtract("dekabr 3",
                    "2017-12-03 00:00:00", "")
        testExtract("5 iyun 2017 ci il axşamı anama zəng etməyi xatırlat",
                    "2017-06-05 19:00:00", "anama zəng etməyi xatırlat")

Is that what you're seeing also?

Are these the only test cases that would rely on the locale getting set or does it point to something else going on?

yeah, These are the cases that rely on locale being set.

@siathon
Copy link

siathon commented Dec 21, 2021

@krisgesling is using the locale setting alright or should I try another way to handle these cases?

@krisgesling
Copy link
Contributor

I'd really like it if we could find another way. But I don't have a suggestion for what that is sorry.

@siathon
Copy link

siathon commented Dec 25, 2021

@krisgesling, Done, no locale setting required for parsing datetime.
575299c

NeonJarbas pushed a commit to NeonJarbas/ovos-lingua-franca that referenced this pull request Jan 19, 2022
NeonJarbas pushed a commit to NeonJarbas/ovos-lingua-franca that referenced this pull request Feb 22, 2022
NeonJarbas pushed a commit to NeonJarbas/ovos-lingua-franca that referenced this pull request Apr 11, 2022
NeonJarbas pushed a commit to NeonJarbas/ovos-lingua-franca that referenced this pull request Apr 11, 2022
extract datetime without setting locale

(cherry picked from commit 575299c)

remove extra comments and imports
fix license year

(cherry picked from commit d449931)

improve az-az duration extract

(cherry picked from commit 6ee8412)

Implemented Azerbaijani support

(cherry picked from commit 0439a80)
JarbasAl pushed a commit to OpenVoiceOS/ovos-lingua-franca that referenced this pull request Apr 11, 2022
extract datetime without setting locale

(cherry picked from commit 575299c)

remove extra comments and imports
fix license year

(cherry picked from commit d449931)

improve az-az duration extract

(cherry picked from commit 6ee8412)

Implemented Azerbaijani support

(cherry picked from commit 0439a80)

Co-authored-by: Siavash Mollayi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA: Yes Contributor License Agreement exists (see https://github.com/MycroftAI/contributors)
Projects
No open projects
Status: Work in progress by author
Development

Successfully merging this pull request may close these issues.

4 participants