Project Data is a way of storing information about projects with its respective versions and their dependencies and properties. A simple example: for Project A, we would store all its valid versions we find in central repository, all the dependencies and properties of all the versions.
The current form of storage is not the most scalable way to store this useful information, as we simply mesh all properties and all dependencies of all versions for a particular project. We want to simplify operations like dependency analysis, but the current storage form doesn't provide any advantage. Since, metadata of any studio project is versioned, we want to start storing project information with respect to versions separately.
We will create a new collection called "versions" in mongo store, which would have decomposed version of project data. The collection would be composed of objects for each version of a particular project and its respective dependencies and properties. We will also have the older collection "project-configurations" present, which would now only store the coordinates (projectId, groupId, artifactId) of the project
- Use the API /migrations/migrateToVersionData, to populate this new "versions" collection.
- Once done, using the /indexes (create indexes if absent) you can create the index for this collection.
- Test all the changes and once sure you can use /migrations/cleanupProjectData, to clean up the project data we store.
Currently, we have an in memory cache for realization of transitive dependencies for a particular version. Other resources use this to retrieve the transitive dependencies and it makes the process performant. Sometimes the cache gets corrupted leading to inconsistencies in the result. It also gets recreated every time we start the server. The solution here is storing the transitive dependencies in the store itself. This would enhance the process of retrieving transitive dependencies by ensuring that there is always a deterministic result.
- Use the API /migrations/calculateDependenciesForVersions/all, to populate a new collection "versionsTemp".
- Use the API /migrations/addTransitiveDependenciesToVersionData, to update the "versions" collection with "versionsTemp" collection delta, which in this case is transitive dependencies.
- After verifying everything works, dop the collection "versionsTemp", using API /collections/{id} PS: The first step ensures that we are not corrupting the old data in cases on rollback.
Versioned Entities are entities stored with the versioned flag being true. They have extra information of versioning in their path. We want to segregate the versioned entities from the entities collection and have separate implementations for both. The driver for this change is to increase the performance of querying entities whether it be versioned or not versioned.
- Use the API /migrations/migrations/deleteVersionedEntities, this will delete all versioned entities from the "entities" collection.
Currently, with Mongo implementation there is limit on how much nesting of an object we can store. Entities sometimes tend to be huge nested objects. Due to Mongo's constraint we can't store such entities and hence it creates a bottleneck in what we can and can't store. In order to store such entities and also make the storage of entities scalable to other databases, we are going to be storing the content as string. With this change we have also introduced three kinds of entity storage which will further help the cause of scalability.
- Use the API /migrations/migrateToStoredEntityData, this will update the entities collection to be identified as one form of entity data.
Currently, we compute latest version whenever it is required. It is computed by fetching all versions for the project coordinates and then find the maximum of all the versions. We wanted to make the APIs using latest version have a better performance. The result is storing latest version to the data itself. We save the computation of this version. We use the refresh handler to keep the latest versions in sync.
- Use the API /migrations/addLatestVersionToProjectData, this will update the project configurations with their respective latest version from versions collection.