* 1422 and 1423 Remove HDFS and Oozie from Menas

* #1422 Fix HDFS location validation

* #1424 Add Menas Dockerfile

* #1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

* #1416 - enceladus on S3:

 - all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running standardization works via:
spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --raw-format json --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/stderr.txt

* #1416 - enceladus on S3 - (crude) conformance works on s3 (s3 std input, s3 conf output)

 0- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running conformance works via:
spark-submit --class za.co.absa.enceladus.conformance.DynamicConformanceJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/conf-log.txt

* ref issue = 1416

* related test cases ignored (issue reference added)

* PR updates

* Merge spline 0.5.3 into aws-poc

* Update spline to 0.5.4 for AWS PoC

* #1503 Remove HDFS url Validation

This is a temporary solution. We currently experiment with
many forms of URLs, and having a regex there now slows us down.

* New dockerfile - smaller image

* s3 persistence (atum, sdk fs usage, ...) (#1526)

#1526 
* FsUtils divided into LocalFsUtils & HdfsUtils
* PathConfigSuite update
* S3FsUtils with tail-recursive pagination accumulation - now generic with optional short-circuit breakOut
TestRunnerJob updated to manually cover the cases - should serve as a basis for tests
* HdfsUtils replace by trait DistributedFsUtils (except for MenasCredentials loading & nonSplittable splitting)
* using final version of s3-powered Atum (3.0.0)
* mockito-update version update, scalatest version update
* S3FsUtilsSuite: exists, read, sizeDir(hidden, non-hidden, reucursive), non-splittable (simple, recursive with breakOut), delete (recursive), version find (simple - empty, recursive)
* explicit stubbing fix for hyperdrive

* Feature/1556 file access PoC using Hadoop FS API (#1586)

* s3 using hadoop fs api
* s3 sdk usage removed (pom, classes, tests)
* atum final version 3.1.0 used
* readStandardizationInputData(... path: String)(implicit ... fs: FileSystem) -> readStandardizationInputData(input: PathWithFs)

* 1554 Tomcat with TLS in Docker container (#1585)

* #1554 Tomcat with TLS container

* #1554 Added envoy config + enabling running unencrypted container

* #1499 Add authentication to /lineage + update spline to 0.5.5

* #1618 - fixes failing spline 0.5.5 integration by providing compatible commons library version. Test-ran on EMR. (#1619)

* #1612 Separation start

* #1612 Updated DAO for spark-jobs

* #1612 Fixed spline integration and schema, removed redundant code

* #1612 Fixed tests, removed unused dependency

* #1612 Added back dependency

* WIP fixing merge issues

* * Merge compiles
* Tests pass
* Depends on ATUM 3.1.1-SNAPSHOT (the bugfix for AbsaOSS/atum#48)

* #1612 Removed Spring from menas-web, enabled building war and static resources. Removed version subpath in menas-web + added theme dependencies in repo

* #1612 Cookies + updated lineage

* * put back HDFS browser
* put back Oozie
* downgraded Spline

* * AWS SDK Exclusion

* #1612 Included HDFSFolder + missing Oozie parts

* * New ATUM version

* * Adding missing files

* #1612 menas-web on nginx container and passing API_URL

* #1612 Working TLS on nginx, resources not included in code

* 1622: Merge of aws-poc to develop brach
* Addressed issues identified by reviewers

* * comments improvement

* 1434 Add new way of serving properties to Docker

* #1612 Building using ui5 + reused /api route

* #1612 Project version

* #713 Add favicon

* #1612 Merges

* #1612 pom parent version

* #1648 Fix war deployment + adding back spline to menas

* #1612 other fixes

* #1612 added pom package.json version sync

* #1612 newline

* #1612 fix version sync + cleaning dist

* 1648 merge to develop

* 1648 merge fix

* 1648 Fixes schema upload

* 1648 Fixes schema registry request

* 1648 pom version

* 1612 add docker build

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #1648 Updating menas-web to 3.0

* #1612 Updated npm project versions + mvn plugin

* #1612 license_check.yml

* #1612 licence check fix

Co-authored-by: Saša Zejnilović <[email protected]>
Co-authored-by: Daniel Kavan <[email protected]>
Co-authored-by: Jan Scherbaum <[email protected]>
Co-authored-by: David Benedeki <[email protected]>

% Conflicts:
%	data-model/src/main/scala/META-INF/MANIFEST.MF
%	menas-web/ui/components/dataset/conformanceRule/MappingConformanceRule/targetSchemaFieldSelector.fragment.xml
%	menas/ui/index.html

…ult values

…velop-ver.30

# Conflicts:
#	menas/src/main/scala/za/co/absa/enceladus/menas/MvcConfig.scala

merging develop into develop-ver3.0 (via  feature/merging-develop-ver.30 branch)

KafkaErrorSenderPluginSuite test fix for SourcePhase.{Standardization, Conformance} capitalization.

* #601 Swagger api docs

…XMLHack removed, the test holds. (#1783)

* #1769 rename menas to rest-api

* #1769 README update and other feedback

* #1733 final version 0.6.0 of spline spark agent used
* #1733 cherrypicked version: commons 0.0.27, atum 3.5.0, spline agent 0.6.0-snapshot
* #1733 PR touchup: maven.shade.version version revert to 3.2.1

* #1770 Rename menas-web to menas

% Conflicts:
%	dao/pom.xml
%	data-model/pom.xml
%	examples/pom.xml
%	menas/pom.xml
%	migrations-cli/pom.xml
%	migrations/pom.xml
%	plugins-api/pom.xml
%	plugins-builtin/pom.xml
%	pom.xml
%	spark-jobs/pom.xml
%	utils/pom.xml

Spline 0.6 integration

% Conflicts:
%	menas/pom.xml
%	menas/ui/index.html
%	pom.xml
%	rest-api/src/main/scala/za/co/absa/enceladus/rest_api/WebSecurityConfig.scala
%	rest-api/src/main/scala/za/co/absa/enceladus/rest_api/controllers/DatasetController.scala

% Conflicts:
%	menas/pom.xml

…elate to rest_api, not menas: `/3rdParty/**` should not be added anymore as it is menas-related)

* Adding back Menas module, that somehow got omitted.

…lop-ver-3

Merging Release 2.23. 0 into develop ver 3

* merge 2.24 into develop-ver-3

* Update for next development version 2.24.0-SNAPSHOT

* Suppress download noise in license check

* Suppress compiler warning of obsolete Java (#1892)

* 1868 statistics with missing counts and datasets missing proprties (#1873)

* 1868 statistics with missing counts and datasets missing proprties

* 1843 Summary page for properties (#1880)

* 1843 Home page with properties,  side panel with missing counts and summary page for properties with tab containing datasets missing that particular property

* Feature/1603 mapping table filtering general (#1879)

* #1603 serde tests for CR and MT DataFrameFilters
(mongo-bson-based serde tests for CR and MT DataFrameFilters, mongo-bson-based serde tests extended for CR with a blank mappingTableFilter)

* #1909 Increase the limit of columns shown in menas column selection

* 1903 Add validation for complex default values in mapping tables on import

* Project config and management updates (#1908)

Project config and management updates
* poc issue template
* CODEOWNERS update
* developers update
* Badges to README.md

* 1881 HyperConformance enceladus_info_version from payload  (#1896)

1881 HyperConformance enceladus_info_version from payload

* #1887 defaultTimestampTimeZone can be source type specific (#1899)

#1887 defaultTimestampTimeZone can be source type specific
* `DefaultsByFormat` extends the `Defaults` trait, being able to read defaults from configuration files
* `DefaultsByFormat` offers further granularity by first checking the format specific setting only then taking the global one
* Basic `GlobalDefaults` are not configuration dependent anymore
* Standardization now user `DefaultsByFormat` for its defaults, where rawFormat is used for format parameter
* Switched to configuration path to be `enceladus.defaultTimestampTimeZone.default` and `enceladus.defaultTimestampTimeZone.[rawFormat]` respectively
* `defaultTimestampTimeZone` is still supported/read as an obsolete fallback
Co-authored-by: Daniel K <[email protected]>

* #1887 defaultTimestampTimeZone can be source type specific (#1916)

#1887 defaultTimestampTimeZone can be source type specific
* rename of the configuration prefix from `enceladus.` to `standardization.`

* #172 Save original timezone information in metadata file (#1900)

* Upgrade of Atum to 3.6.0
* Writing the default time zones for timestamps and dates into _INFO file

* #1894 `HadoopFsPersistenceFactory` - adding Spline S3 write support (#1912)

* #1894 Spline S3 support via custom persistence factory `HadoopFsPersistenceFactory`.
Co-authored-by: David Benedeki <[email protected]>

* Update versions for release v2.24.0

* Update for next development version 2.25.0-SNAPSHOT

* #1926 Add executor extra java opts to helper scripts

* #1931 Add switch for running kinit in helper scripts

* #1882 Update Cobrix dependency to v.2.3.0

* #1882 Remove explicit "collapse_root" since it is the default since Cobrix 2.3.0

* #1882 Update Cobrix to 2.4.1 and update Cobol test suite for ASCII files.

* #1882 Bump up Cobrix version to 2.4.2.

* #1927 Spline _LINEAGE and Atum _INFO files permission alignment (#1934)

* #1927 - testing setup: set both spline _LINEAGE and atum _INFO to hdfs file permissions 733 -> the result on EMR HDFS was 711 (due to 022 umask there) -> evidence of working

* #1927 - cleanup of test settings of 733 fs permissions

* #1927 Atum final version 3.7.0 used instead of the snapshot (same code)

* #1927 comment change

* #1927 - default 644 FS permissions for both _INFO and _LINEAGE files.

* 1937 limit output file size (#1941)

* 1937 limit output file size

* 1937 limit output file size

* 1937 renamings + constants

* 1937 more conditions

* 1937 rename params

* 1937 feedback + script params

* 1937 more feedback

* 1937 final feedback

* #1951: Windows Helper scripts - add missing features
* `ADDITIONAL_JVM_EXECUTOR_CONF`
* Kerberos configuration
* Trust store configuration
* kinit execution option
* `--min-processing-block-size` & `--max-processing-block-size`
* logo improvement

* * --min-processing-block-size -> --min-processing-partition-size
* --max-processing-block-size -> --max-processing-partition-size

* #1869: SparkJobs working with LoadBalanced Menas (#1935)

* `menas.rest.retryCount` - configuration, how many times an url should be retried if failing with retry-able error implemented
* `menas.rest.availability.setup` - configuration, how the url list should be handled
* _Standardization_, _Conformance_ and _HyperConformance_ changed to provide retry count and availability setup to Dao, read from configuration
* `ConfigReader` enhanced and unified to read configurations more easily and universally
* Mockito upgraded to 1.16.42

Co-authored-by: Daniel K <[email protected]>

* Feature/1863 mapping table filtering (#1929)

* #1863 mapping cr & mt fitler successfully reuses the same fragment (both using the same named model)
 - todo reuse validation, reuse manipulation methods

* #1863 FilterEdit.js allows reusing filterEdit TreeTable logic between mCR and MT editings

* #1863 mCT editing validation enabled (commons from FilterEdit.js)

* #1863 mCT datatype hinting hinting enabled (commons from DataTypeUtils.js)

* #1863 mCR/MT edit dialog default width=950px, some cleanup
* #1863 bugfixes: directly creating MT with filter (fix on accepting the field), UI fix for MT filter model initialization

* #1863 npm audit fix

* #1863 bugfix: adding new mCR (when no edit MCR dialog has been opened yet) did not work - fixed

* #1863 selecting mapping column from MT schema works (for all schema levels) for edit. TODO = Schema type support

 #1863 mCR - schema-based columns suggested for filter, value types filled in silently during submit, too.

* #1863 bugfix: empty MT - schema may be empty

* #1863 bugfix: removing a filter left a null node - cleanup was needed (otherwise view would fail)
logging cleanup

* #1863 select list item now shows valueType as additionalText, cleanup

* #1863 nonEmptyAndNonNullFilled - map->filter bug fixed.

* #1863 typo for null filter

Co-authored-by: David Benedeki <[email protected]>

* Update versions for release v2.25.0

* [merge] build fix

* [merge] npm audit fix

* [merge] npm audit fix

* [merge] buildfix (menas->rest_api packaging fix)

* [merge] review updates

Co-authored-by: David Benedeki <[email protected]>
Co-authored-by: Saša Zejnilović <[email protected]>
Co-authored-by: David Benedeki <[email protected]>
Co-authored-by: Adrian Olosutean <[email protected]>
Co-authored-by: Ruslan Iushchenko <[email protected]>

…e contours, empty DatasetControllerV3 to test

 - login allowed at /api/login & /api-v3/login
 - v2/v3 BaseRestApiTest distinquished

…st cases added

* #2022 spark-commons

* New class `TZNormalizedSparkTestBase` used over all tests instead of `SparkTestBase`

…der and no content -> IT updated

… IT updated

…version, includes a sample CR)

…st and post-import + IT to prove correct behavior

…mport + IT to prove correct behavior - fix with common location processing with segment stripping (+normalization)

… GET datasets/{name}/{version}

…/audit-trail added

…ion} now works for # or 'latest' (IT = regression test)

…{name}/{version} and /{name}/latest - improved

…- supports latest for as version-expression, impl for datasets improved by actual existence checking + IT test cases for non-existing/non-latest queries

…properties - supports latest for as version-expression; get impl is unvalidated, put impl checks validity

 - login is now common, under /api/login for both v2 and v3 (did not work previously)

…properties - supports latest for as version-expression; get impl is unvalidated, put impl checks validity - extended for different validation cases

 - login is now common, under /api/login for both v2 and v3 (did not work previously)

…ture -> CompletableFuture (mistake reverted)

… added. IT mostly adjusted, but there are todos

 - DatasetServiceV3 introduced to carry difference in behavior to DatasetService. original entity validation has been divided into create-validation and regular-entity validation.
 - buildfix for VersionedModelServiceTest

…sion}` in proper validations

…asets/dsName/version/rules, GET datasets/dsName/version/rules/# + IT

 - v2 ignores this added information
 - needs

…api-v3-schema-etc

…api-v3-schema-etc

…defaults

…ts + IT

…api-v3-schema-etc

#2050: Unifying project space with GitHub defaults
* docs -> documentation
* Milestones are obsolete

…l, .../from-file

VersionedModelControllerV3 now requires admin rights on specific endpoints (create, edit, import, delete) - TODO reflect in other datatypes

…d in DatasetControllerV3, MappingTableControllerV3, SchemaControllerV3. ITs fixed, but 403 tests missing

… 403 cases for all types

…y fields + PUT/POST with schema with empty fields -> 400 validation error, too. + ITs reflecting this (this reqiured parallel SchemaServiceV3)

…s to the implementation of V3 alongside of V2

… (PUT .../{name}) implemented. ITs cover dataset disabling

 - BaseRestApiTest omits entity for Delete calls + reflected in various places

…name}/used-in & /{name}/{version}/used-in). The former is used in disable checking.

IT cases added for Schema usedIns, Datasets with disabled entities

… test cases.

…api-v3-schema-etc

…3-delete-recreate

…xtensible via `updateFields` method - V3 uses this to reflect schema fields (actual name of the field :D).

…3-delete-recreate

…s for usedIn and delete added.

Custom alternative:
```scala
//  /**
//   * Finds references of MappingTable in Dataset's mapping conformance rules
//   * @param mtName
//   * @return
//   */
//  def findRefByMappingTableInMappingConformanceRules(mtName: String, version: Option[Int]): Future[Seq[MenasReference]] = {
//    /* The gist of the find query that this method is based on; testable in a mongo client
//      { $and : [
//        {...}, // non disabled
//        {"conformance": {"$elemMatch": {$and : [
//          {"_t": "MappingConformanceRule"},
//          {"mappingTable": "AnotherAwesomeMappingTable"} // mappingTableVersion not needed for match, but will be reported as part of the MenasReference
//        ]}}}
//      ]}
//     */
//    val versionFilter = version.fold(Filters.and() /*empty filter*/ )(ver => Filters.eq("mappingTableVersion", ver))
//
//    val filter: Bson = Filters.and(
//      getNotDisabledFilter,
//      Filters.elemMatch("conformance", Filters.and(
//        Filters.eq("_t", "MappingConformanceRule"),
//        Filters.eq("mappingTable", mtName),
//        versionFilter
//      ))
//    )
//
//    collection
//      .find[MenasReference](filter)
//      .projection(fields(include("name", "version"), computed("collection", collectionBaseName)))
//      .sort(Sorts.ascending("name", "version"))
//      .toFuture()
//  }
```

… between None and Some(Seq.empty) in various places, V3 consistently returns None for no references groups -> reflected in V3 ITs. + cleanup

…vice and DatasetServiceV3 with renaming.

VersionList removed in favor of VersionedSummary everywhere.
- explanatory comments

…api-v3-schema-etc

…3-delete-recreate

% Conflicts:
%	rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/v3/DatasetServiceV3.scala

…rt mapping removed -> reflected in IT; small updates

… removed, some ITs and comments added