Embed the value objects

[mahalakshme]

This story is in sequence to this spike. Model changes pushed to realm-embed branch. Also use the test in the commit in the spike.

Acceptance criteria:

• Entities to update/migrate
  -- Format
  -- KeyValue
  -- StringKeyNumericValue
  -- Point
  -- ChecklistItemStatus
  -- Decision - would probably be removed before this story is played since its deprecated
  -- Observation
  -- VisitScheduleInterval - would probably be removed before this story is played since its deprecated

• Not to be done because these are top level object, not referred by any other entity
  -- EntityQueue
  -- ConfigFile - would probably be removed before this story is played since its deprecated
  -- BeneficiaryModePin
  -- Extension
  -- UserDefinedIndividualProperty - would probably be removed before this story is played since its deprecated
  -- VisitScheduleConfig - would probably be removed before this story is played since its deprecated

• migration is required like (in realm-embed model branch). schemaVersion number may have changed when the story is played.

• deleting orphans can be done by changing the schema name. recommend doing it only for observations and Point. others are small entities. The change is present in the spike branch.

• Calls to load emebdded object types directly have to be removed (e.g. db.object(Observation.schema.name) - file link). How to handle for the ones usage is found.
  -- Observation in PruneMedia (change code to do this per parent entity)

• Also calls like these to now delete embedded objects should be removed

• it takes about 30 seconds to migrate observations in large database. Loading white screen is shown, we should modify this to say that 'Upgrading data, Please do not close the screen.'
  -- This should happen when upgrading database after fast sync as well. Say fast sync is setup with schema version 180 and this change is in schema version 181. So after restoring db when executing this line migrations run.

Additional observations from spike:

• add embedded as following example

name: "Format", embedded: true, properties: { regex: "string", descriptionKey: "string" }

• embedded object type can be used multiple times
• cannot query embedded objects directly from database (even though it has a schema name). can query using the patch of embedded object

sample test cases:

• When a parent object, say individual is deleted, registration location, observations, etc., should also get deleted.
• When an observation of an individual is updated, the old observation that now doesn't have a parent should be deleted.
• Subject Migration test cases to be validated

[mahalakshme]

@petmongrels have added deleting orphans to be run for Point also since for one of the users I tested there were 24L entries in fast sync vs actual 17k with fresh sync. Let me know if you think otherwise.

[mahalakshme]

@petmongrels if this card, played before this, need not handle cases where I ve mentioned 'would probably be removed before this story is played since its deprecated' in the above description.

[petmongrels]

marked this as high risk. #1241
some Schema maybe present in dumps, but not in our current code. plus unknown unknowns.

Testing with older dumps for multiple organisations may reveal issues apart from regular regression that will be needed of the mobile app.

[himeshr]

Test cases successfully validated:

1. Login using 6.1 apk and then upgrade to master apk
2. Direct login without fast sync with master apk
3. Login using 6.1 apk and then setup fast sync. Logout and clear data and login using master apk, which restores fast sync
4. Modifying observation and ensuring no dangling entities
5. Deleting entity and ensure no dangling entities
6. Subject migration of entities and no dangling entities