New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

API Substitute Zend_Locale / Zend_Date with php-intl #6607

Merged

chillu merged 5 commits into silverstripe:master from open-sausages:pulls/4.0/i18n-locale

Feb 15, 2017

Contributor

tractorcow commented Feb 8, 2017 •

edited by dhensby

Loading

Fixes #6194

Requires:

Note: This PR removes about 1/4 of a million lines of code. Yay. :D

Includes a few other enhancements:

i18nTextCollector supports __CLASS__
Update FormField with a setSubmittedValue to differentiate between loading values from the database and loading values from user-submitted field values.
Add Resettable interface in place of having to manually invoke reset() on explicit classes between test resets.

Some minor functionality around currency names could not be ported across, and have been removed.

tractorcow added affects/v4 type/api-break type/enhancement labels

This was referenced Feb 8, 2017

RFC Replace Zend_Locale with new i18n backend #6194

Closed

Remove unnecessary tinymce config silverstripe/silverstripe-reports#59

Merged

Update CMS for new i18n API silverstripe/silverstripe-cms#1734

Merged

Update to use php-intl instead of zend_date silverstripe/silverstripe-testsession#43

Merged

Contributor Author

tractorcow commented Feb 8, 2017

Some immediate feedback is that Resettable is a bad design pattern. Perhaps we should address this at a later date with better locale caching.

tractorcow force-pushed the pulls/4.0/i18n-locale branch 3 times, most recently from 294070f to a9d39d2 Compare

February 9, 2017 02:08


          API Substitute Zend_Currency with NumberFormatter based solution

029a8b9

API Substitute Zend_Locale with Locale / NumberFormatter
API Substitute Zend_Date with IntlDateFormatter
API Added DBTIme::Nice12, FormatFromSettings
API Added Short() method to DBDate / DBTime / DBDatetime
API Add Date::getTimestamp()
API Added setSubmittedValue api for FormField
API Add second arg to base FormField::setValue()
API Major refactor of i18n into component data parts
API Implement Resettable interface to reset objects between tests
ENHANCEMENT Changed DBField::create_field return type to `static` to support better type hinting
ENHANCEMENT i18nTextCollector supports __CLASS__

tractorcow force-pushed the pulls/4.0/i18n-locale branch from a9d39d2 to 029a8b9 Compare

February 9, 2017 02:29

chillu requested changes

View reviewed changes

Member

chillu left a comment

First round of review (got up to Form.php)

docs/en/04_Changelogs/4.0.0.md Outdated

               * `ChangeSet` and `ChangeSetItem` have been added for batch publishing of versioned dataobjects.
               * `DataObject.table_name` config can now be used to customise the database table for any record.
               * `DataObjectSchema` class added to assist with mapping between classes and tables.
+              * `DBMoney` values are now treated as empty only Amount is null. Values without Currency

Member

chillu Feb 13, 2017

The first sentence doesn't make sense ;)

Contributor Author

tractorcow Feb 14, 2017

Updated to

* `DBMoney` values are now treated as empty only if `Amount` field is null. If an `Amount` value
  is provided without a `Currency` specified, it will be formatted as per the current locale.

docs/en/04_Changelogs/4.0.0.md Outdated

+                See http://userguide.icu-project.org/formatparse/datetime.
+              * getISOFormat() added which returns the standard date/time ISO 8601 pattern in CLDR format.
+              * Dates passed in m/d/y format will now raise a notice but will be parsed.
+                Dates passed to constructors should follow ISO 8601 (y-m-d).

Member

chillu Feb 13, 2017

You should use CLDR format strings to describe the dates here, for consistency: YYYY-MM-dd. Same with the line above.

docs/en/04_Changelogs/4.0.0.md Outdated

+              * `Format()` method now use CLDR format strings, rather than PHP format strings.
+                See http://userguide.icu-project.org/formatparse/datetime.
+              * getISOFormat() added which returns the standard date/time ISO 8601 pattern in CLDR format.
+              * Dates passed in m/d/y format will now raise a notice but will be parsed.

Member

chillu Feb 13, 2017

"Dates passed with leading months (MM/dd/YYYY)..."

docs/en/04_Changelogs/4.0.0.md Outdated

+              The below methods have been added or had their functionality updated to `DBDate`, `DBTime` and `DBDatetime`
+              * `getTimestamp()` added to get the respective date / time as unix timestamp (seconds since 1970-01-01)
+              * `Format()` method now use CLDR format strings, rather than PHP format strings.

Member

chillu Feb 13, 2017

Add some common examples for how they're different: "From d/m/Y H:i:s to dd/MM/y HH:mm:ss (Example: 31/07/2017 23:01:01)"
Link "PHP format" to http://php.net/manual/en/function.date.php
Link "CLDR" to http://userguide.icu-project.org/formatparse/datetime instead of writing out the URL (shorter, easier to review). Link

docs/en/04_Changelogs/4.0.0.md Outdated

+              * getISOFormat() added which returns the standard date/time ISO 8601 pattern in CLDR format.
+              * Dates passed in m/d/y format will now raise a notice but will be parsed.
+                Dates passed to constructors should follow ISO 8601 (y-m-d).
+              * 2-digit years will raise a notice.

Member

chillu Feb 13, 2017

"2-digit years in setValue() will raise a notice, since they're ambiguous without additional formatting hints"

docs/en/04_Changelogs/4.0.0.md Outdated

+              `DBTime` specific changes:
+              * Added `DBTime::FormatFromSettings`
+              * Added `DBTime::Nice12`

Member

chillu Feb 13, 2017

Shouldn't we remove Nice24 rather than adding Nice12, given that you do all of this via custom formatting now? The original purpose for this stuff was the lack of passing complex arguments via SSViewer, but that's since been resolved.

Contributor Author

tractorcow Feb 14, 2017

Sure. :)

docs/en/04_Changelogs/4.0.0.md

               * `$message` second constructor parameter is removed. Constructor only accepts `$result`,
                 which may be a string, and optional `$code`
+              New `DatetimeField` methods replace `getConfig()` / `setConfig()`:

Member

chillu Feb 13, 2017

What happened with datavalueformat?

Contributor Author

tractorcow Feb 13, 2017

It's fixed to iso format. No more custom format for internal values.

Contributor Author

tractorcow Feb 14, 2017 •

edited

Loading

Added * ``datavaluefield`` config is removed as internal data value is now fixed to ISO 8601 format

docs/en/04_Changelogs/4.0.0.md Outdated

                 for all DataObject subclasses, rather than just the basename without namespace.
               * i18n key for locale-respective pluralisation rules added as '.PLURALS'. These can be configured
                 within yaml in array format as per [ruby i18n pluralization rules](http://guides.rubyonrails.org/i18n.html#pluralization).
+              * `i18n.all_locales` config moved to `Locales.locales`

Member

chillu Feb 13, 2017

All of these should mention the correct namespace, otherwise it's harder to fix via copy/paste.

src/Core/Resettable.php

+               * Designed so that tests can automatically flush these objects between runs in lieu
+               * of a real change.
+               */
+              interface Resettable

Member

chillu Feb 13, 2017

Shouldn't we move this stuff to explicit registration via YAML config? My understanding was that ClassInfo::implementorsOf() is on it's way out, because it forces us to parse all PHP files on a flush (which is no longer required for autoloading thanks to PSR-7). The interface Resettable is fine, but I wouldn't use ClassInfo::implementorsOf('Resettable') to retrieve those classes.

Contributor Author

tractorcow Feb 14, 2017

That's a problem I'm not sure we can solve in this pull request.

Member

chillu Feb 14, 2017

My point is that introducing more use cases is taking it in the wrong direction. But you're right, this is best addressed separately - the interface itself is good, we might just move all those to explicit registration at some point.

Member

sminnee Feb 14, 2017

My thinking is that we should introduce some kind of Application object that contains any in-memory caches and other objects that we might want to re-set, and have our tests create new Application objects. But it's a separate PR.

src/Forms/DateField.php

                    */
-                  protected $config;
+                  protected $showCalendar = false;

Member

chillu Feb 13, 2017

One of the reasons I've avoided elevating these settings to class properties is that I didn't want to lock in the API to that extent - it's a bit like using FormField->setAttributes() to influence client-side react component presentation.

So rather than increasing the API surface of DateField, what do you think about separating out DateFieldSeparated extends DateField and DateFieldJQuery extends DateField? Then we can set the DateField injector service name to resolve toDateFieldJQuery by default, in order to get correct behaviour for built-in features like DBDate->scaffoldFormField()

Contributor Author

tractorcow Feb 14, 2017

I've split out DateFieldSeparated, although I'm a bit hesitant to make a jquery-specific class which may limit usability (e.g. in react-formschema environments).

Member

chillu Feb 15, 2017

OK, I've reinstated format-based ordering and custom separators now

chillu requested changes

View reviewed changes

Member

chillu left a comment

Second round (up to DBDate.php)

src/Forms/FormField.php

                    * @return $this
                    */
-                  public function setValue($value)
+                  public function setValue($value, $data = null)

Member

chillu Feb 13, 2017

You should note the change of method signature in the changelog, since it'll only cause a Strict notice when the field is actually used (meaning you hit a page with this particular form on it). Also, recommend setSubmittedValue() over setValue() there.

The forms.md docs are slightly wrong now:

The data value of the FormField submitted is not passed into validate. It is stored in the value property through the setValue method.

Contributor Author

tractorcow Feb 14, 2017

Not sure where forms.md is; I'll have a look at refreshing the forms docs dir though.

Contributor Author

tractorcow Feb 14, 2017

Found and fixed. :)

src/Forms/MemberDatetimeOptionsetField.php Outdated

+                  public function setValue($value, $data = null)
+                  {
+                      if (is_array($value)) {
+                          throw new InvalidArgumentException("Invalid value");

Member

chillu Feb 13, 2017

Hint that it's "Invalid because it's an array, expected X"

Contributor Author

tractorcow Feb 14, 2017

throw new InvalidArgumentException("Invalid array value: Expected string");

src/Forms/MoneyField.php Outdated

-                      if ($allowedCurrencies) {
-                          $field = new DropdownField(
+                      if (count($allowedCurrencies) === 1) {
+                          // Dropdown field for multiple currencies

Member

chillu Feb 13, 2017

Inaccurate comment

Contributor Author

tractorcow Feb 14, 2017

Oh dear, will fix!

src/Forms/MoneyField.php Outdated

+                      // Convert string to array
+                      // E.g. `44.00 NZD`
+                      if (is_string($value) &&
+                          preg_match('/^(?<amount>[\\d\\.]+)( (?<currency>\w{3}))?$/i', $value, $matches)

Member

chillu Feb 13, 2017

This hardcodes assumptions about decimal point formatting (Germans use a comma instead of a dot). Other countries will place the currency before the amount, or shorten it with a symbol. We didn't do string parsing before - why has it become necessary now? Any why did you remove support for the array notation which was designed to solve this issue? (consistent array payloads in form submission, rather than inconsistent string payloads)

Member

chillu Feb 13, 2017

Oh right, setValue() is now only supposed to be used for hydration from the database, and setSubmittedValue() still supports arrays. And the assumption here is that hydration can either be from a DBMoney composite database fields (two columns), or a single string database field (which is a new feature you've added). This means we devs can control the formatting of the string field (not defined by user input), but it's still a bit odd to support this when we have a superior way with the more normalised DBMoney? It also means you can't properly prepopulate a field with values, without casting into DBMoney first.

Contributor Author

tractorcow Feb 14, 2017

It also means you can't properly prepopulate a field with values, without casting into DBMoney first.

You can pass in an array with Amount and Currency keys.

Contributor Author

tractorcow Feb 14, 2017

Oh dear, I'm throwing an exception... I'd intended to support that array format, sorry!

Contributor Author

tractorcow Feb 14, 2017

Changed error case to

} elseif (!is_array($value)) {
    throw new InvalidArgumentException("Invalid currency format");
}

src/Forms/MoneyField.php

    
                  public function setLocale($locale)

                  {

                      $this->_locale = $locale;

                      $this->fieldAmount->setLocale($locale);

Member

chillu Feb 14, 2017

This breaks for readonly versions of this field:

[Emergency] Uncaught BadMethodCallException: Object->__call(): the method 'getLocale' does not exist on 'SilverStripe\Forms\NumericField_Readonly'

Contributor Author

tractorcow Feb 14, 2017

I've modified this so that fieldAmount remains a NumericField, but is assigned setReadonly(true) instead of being type-shifted.

src/Forms/DateField.php Outdated

-                      } // Default text input field
-                      else {
-                          $html = parent::Field();
+                      $valArr = $this->iso8601ToArray($this->Value());

Member

chillu Feb 14, 2017

This breaks when locale isn't en_us, since Value() uses iso8601ToLocalised(), but iso8601ToArray() expects ISO

Contributor Author

tractorcow Feb 14, 2017

Oops! Can fix with $this->dataValue() instead .:)

src/Forms/NumericField.php

		*/
		public function performReadonlyTransformation()

Member

chillu Feb 14, 2017

Somehow the readonly version doesn't respect setLocale('de') or setScale(4). The disabled version is fine, weird. Example from a slightly modified frameworktest:

Contributor Author

tractorcow Feb 14, 2017

Deleting NumericField_Readonly helped. :)

src/Forms/NumericField.php

                    * @return $this
                    */
-                  public function setLocale($locale)
+                  public function setScale($scale)

Member

chillu Feb 14, 2017

Where did you get the "scale" terminology from? The ICU docs behind PHP's NumberFormatter refer to this as the "significant digit": http://www.icu-project.org/apiref/icu4c/classDecimalFormat.html#details

Contributor Author

tractorcow Feb 14, 2017 •

edited

Loading

Nope nothing to do with significant digits. scale is number of digits to the RIGHT of the decimal point. significant digits means significant digits from the left of the number. E.g. 1340.0 is 1335.00 to 3 significant digits. What i want is the number of digits (including zeros) after ..

Contributor Author

tractorcow Feb 14, 2017

I can't find the correct term, to be honest. Closest reference I could find is https://msdn.microsoft.com/en-us/library/ms190476.aspx

Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

Member

chillu Feb 14, 2017

Actually, both left/right digits are "significant" - quoting https://en.wikipedia.org/wiki/Significant_figures#Rounding_and_decimal_places

Trailing zeros in a number containing a decimal point are significant. For example, 12.2300 has six significant figures: 1, 2, 2, 3, 0 and 0

But yeah, you're correct that "significant digits" is the wrong term here. I'm used to "decimal places", but since you've documented it well, I'm also happy with "scale"

Contributor Author

tractorcow Feb 14, 2017

Yes, significant figures are counted FROM the left, but potentially including digits on the right. Not ONLY whole number values. :)

src/ORM/FieldType/DBDate.php Outdated

    
            @@ -271,18 +309,23 @@ public function Rfc822()
          
                  public function Rfc2822()

                  {

                      if ($this->value) {

                          return date('Y-m-d H:i:s', strtotime($this->value));

                          return date('Y-m-d H:i:s', $this->getTimestamp());

Member

chillu Feb 14, 2017

Shouldn't we use getFormatter() consistently here, rather than relying on date() for some of the features? It'll be confusing to refer back to two different symbol systems (date() and ICU)

Contributor Author

tractorcow Feb 14, 2017

I was too lazy to re-write the strings...

Member

chillu Feb 14, 2017

Fair enough, there's still around two dozen occurrences of date() in core - low priority, not a merge blocker.

chillu requested changes

View reviewed changes

Member

chillu left a comment

Third and final part of the review

src/ORM/FieldType/DBDate.php Outdated

+                      // US date format with 4-digit year first
+                      if (preg_match('#^(?<year>\\d{4})/(?<day>\\d+)/(?<month>\\d+)(?<time>.*)$#', $value, $matches)) {
+                          trigger_error(
+                              "Implicit y/d/m conversion. Use " . self::ISO_DATE . " to prevent this notice.",

Member

chillu Feb 14, 2017

Add value to error message (since y/d/m is more of a guess at this point, could well be y/m/d)

Contributor Author

tractorcow Feb 14, 2017

who would do such a thing though? =(

Contributor Author

tractorcow Feb 14, 2017

I'm just going to stop falling back and fail any non-iso 8601 format. Too much maintenance.

Contributor Author

tractorcow Feb 14, 2017

DBDate / DBDatetime now fail if passed not-iso8601 date as values

src/ORM/FieldType/DBDate.php Outdated

-                      $pastDate = mktime(0, 0, 0, $fmonth, $fday, $fyear);
-                      $curDate = mktime(0, 0, 0, date('m'), date('d'), $fyear);
+                      // US date format without 4-digit year first: assume m/d/y
+                      if (preg_match('#^(?<month>\\d+)/(?<day>\\d+)/(?<year>\\d+)(?<time>.*)$#', $value, $matches)) {

Member

chillu Feb 14, 2017

How do you know that it's m/d/y here? Could be NZ date format with a short year (d/m/y), e.g. 31/07/17. I'd change the error message to "Ambiguous short year format detected. Use <ISO_DATE> to prevent this notice". Also, add the value to the error message.

Contributor Author

tractorcow Feb 14, 2017

Slashes = US date.

Contributor Author

tractorcow Feb 14, 2017

http://php.net/manual/en/function.strtotime.php#100144

The best way to compensate for this is by modifying your joining characters. Forward slash (/) signifies American M/D/Y formatting, a dash (-) signifies European D-M-Y and a period (.) signifies ISO Y.M.D.

Contributor Author

tractorcow Feb 14, 2017

I'm going to simplify the hell out of this.

src/ORM/FieldType/DBDate.php Outdated

    
                   */

                  public static function past_date($fmonth, $fday = 1, $fyear = null)

                  protected function explodeDateString($value)

Member

chillu Feb 14, 2017

Why did you replace the new DateTime($value) based parsing with regexes here? DateTime is based on strtotime(), which supports more formats than you cover here. Is this an attempt to warn when passing invalid or ambiguous formats like two-digit years, and narrow down edge cases? Seems reasonable. but will require a changelog entry. The PHPDoc should also explain why it's not using strtotime()

Contributor Author

tractorcow Feb 14, 2017

As with DateField, I'm not trying to support arbitrary custom formats. DBDate / DBDatetime are intended to load records from the database, and I've chosen a restrictive set of date formats based on this restriction.

If you are passing in raw user into into DBDate / DBDatetime then something is wrong.

Member

chillu Feb 15, 2017

We've discussed this a bit more on Slack, and Damian confirmed that the intent was to narrow down supported cases. Given this is populated from the database or somewhat filtered/sanitized user input, that's OK

src/i18n/Data/Intl/IntlLocales.php

+                      if (!$lang || !$region) {
+                          return false;
+                      }
+                      return strcasecmp($lang, $region)

Member

chillu Feb 14, 2017

Is multiplication really what you want here? Seems like a roundabout way of expressing a conditional

Contributor Author

tractorcow Feb 14, 2017

I can make it more explicit.

src/i18n/TextCollection/i18nTextCollector.php

                       foreach ($tokens as $token) {
                           if (is_array($token)) {
                               list($id, $text) = $token;
+                              // Check class

Member

chillu Feb 14, 2017

What are these text collector changes about?

Contributor Author

tractorcow Feb 14, 2017

I'll add to 4.0.0.md. This is so that you can use _t(__CLASS__.'.SOMEKEY') instead of the FQN.

Member

sminnee Feb 14, 2017

i18nTextCollector could probably use a rewrite to make use of php-parser. Separate optional PR, though.

src/i18n/i18n.php

-                      }
-                      return $paths;
+                      return static::getData()->scriptDirection($locale);

Member

chillu Feb 14, 2017

Why didn't you move this to Locale as well?

Contributor Author

tractorcow Feb 14, 2017

Because i18n has a global template provider that needs this method.

Contributor Author

tractorcow commented Feb 14, 2017

I've added another commit with changes requested. Will fix tests tomorrow.


          API Create SeparatedDateField

014f0d2

API Restrict allowed values parsed via DBDate::setValue
API Remove NumericField_Readonly
API Remove DBTime::Nice12 / Nice24

tractorcow force-pushed the pulls/4.0/i18n-locale branch from 6e08a18 to 014f0d2 Compare

February 14, 2017 22:08

Contributor Author

tractorcow commented Feb 14, 2017

Ok, if tests pass this is good to merge.

Contributor Author

tractorcow commented Feb 14, 2017

ok, all non-cms tests passing. CMS tests will pass with cms pr merged.

sminnee reviewed

View reviewed changes

src/Forms/HTMLEditor/TinyMCEConfig.php

+                   * @config
+                   * @var array
+                   */
+                  private static $tinymce_lang = [

Member

sminnee Feb 14, 2017

It seems like this static could only list the exceptions to the rule strtok($input, '_') and be a lot shorter.

sminnee approved these changes

View reviewed changes

Member

sminnee left a comment

Assuming Ingo's okay with this, I am too :-)

src/i18n/TextCollection/i18nTextCollector.php

                       foreach ($tokens as $token) {
                           if (is_array($token)) {
                               list($id, $text) = $token;
+                              // Check class

Member

sminnee Feb 14, 2017

i18nTextCollector could probably use a rewrite to make use of php-parser. Separate optional PR, though.


          Fixed DateFieldSeparated docs

a9f2e9e

chillu reviewed

View reviewed changes

docs/en/02_Developer_Guides/03_Forms/Field_types/02_DateField.md Outdated

               ## Separate Day / Month / Year Fields
-              The following setting will display your DateField as three input fields for day, month and year separately. HTML5
-              placeholders 'day', 'month' and 'year' are enabled by default.
+              To display separate input fields for day, month and year separately you can use the `DateFieldSeparated` subclass`.

Member

chillu Feb 15, 2017

This references DateFieldSeparated rather than SeparatedDateField - I've fixed this in a follow-up commit.

chillu reviewed

View reviewed changes

src/Forms/SeparatedDateField.php Outdated

+                      }
+                      // Join all fields
+                      // @todo custom ordering based on locale

Member

chillu Feb 15, 2017

I'm just implementing this - it's a needless feature regression compared to 3.x


          Reinstated separated DateField functionality

80723c0

chillu approved these changes

View reviewed changes


          Mark setShowCalendar() as experimental

b7bed18

chillu merged commit 2805a13 into silverstripe:master

chillu deleted the pulls/4.0/i18n-locale branch

February 15, 2017 21:56

Contributor

dhensby commented Feb 15, 2017

whoop

Member

chillu commented Feb 15, 2017

Created a follow up issue with #6626

chillu mentioned this pull request

[2010-03-09] Use Zend_Locale to replace/simplify many i18n routines. #1914

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

affects/v4 type/api-break type/enhancement