diff --git a/.github/workflows/gradle.yml b/.github/workflows/gradle.yml new file mode 100644 index 0000000..7cf2780 --- /dev/null +++ b/.github/workflows/gradle.yml @@ -0,0 +1,26 @@ +# This workflow will build a Java project with Gradle +# For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-gradle + +name: Java CI with Gradle + +on: + push: + branches: [ develop ] + pull_request: + branches: [ develop ] + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - name: Set up JDK 1.8 + uses: actions/setup-java@v1 + with: + java-version: 1.8 + - name: Grant execute permission for gradlew + run: chmod +x gradlew + - name: Build with Gradle + run: ./gradlew build diff --git a/README.md b/README.md index b0d05af..642fc5d 100644 --- a/README.md +++ b/README.md @@ -2,19 +2,74 @@ ## Connector Description -Drupal8 Connector fetches the entries of a json file. +The Drupal Connector fetches all the content that is available on the URL provided from Drupal Module. Using an algorithm that takes all the links from every page, all the content from those pages is going to be parsed and normalised in a Fusion 5 Server instance, using the SDK connector. + +## System Diagram +The System Diagram presents the 3 components and how are connected. Fusion is the component where the Java Connector will be uploaded as a plugin. This connector plugin will then be used as a datasource in _Index Workbench_. One of the properties that this connector expects is a URL coming from Drupal Module, from where all the content will be taken recursively and get indexed. + + ![diagram](diagram.png) ## Quick start -1. Clone the repo: +### Clone the repo: +``` +git clone https://github.com/lucidworks/drupal-connector.git +``` + +### Build the project: ``` -git clone https://github.com/lolaent/fusion-connector-java.git cd drupal8 ./gradlew clean build assemblePlugin ``` -2. This produces the zip file, named `drupal8.zip`, located in the `build` directory. -This artifact is now ready to be uploaded directly to Fusion as a connector plugin. + This produces the zip file, named `drupal8.zip`, located in the `build` directory. +This artifact is now ready to be uploaded directly into Fusion Server as a connector plugin. + +### Connector properties +This connector is using the `connector-plugin-sdk` version `2.0.1` which is compatible with Fusion Server 5. + +#### Properties required +1. **_Drupal URL_** - the link from where this connector takes all the content. +2. **_Username_** - the username used to login into drupal to be able to fetch a specific type of content. There are different roles for users defined in that module. +3. **_Password_** +4. **_Login Path_** - the path used to the login request - ```defaultValue = "/user/login"``` +4. **_Logout Path_** - the path used to the logout request - ```defaultValue = "/user/logout"``` +5. **_Entry Path_** - this entry indicates the page from where fetching the content begins - ```defaultValue = "/en/fusion"``` + +#### Properties added in MANIFEST.MF +``` +Plugin-Id: com.lucidworks.fusion.connector +Plugin-Type: connector +Plugin-Provider: Lucidworks +Plugin-Version: 1.0-SNAPSHOT +Plugin-Connectors-SDK-Version: 2.0.1 +Plugin-Class: com.lucidworks.fusion.connector.ConnectorPlugin +``` + +### The Fetcher +The **_JsonContentFetcher_** class provides methods that define how data is fetched and indexed for Fusion Server. The data is fetched from Drupal using the OkHttp Client to call the request. But before the actual request is done to get all the content a login request is needed. There are different types of users that can see the entire content or just a particular part from it. + From the login response the header _Set-Cookie_ is taken and used as a header for the next requests. + +### Crawling process +The **_DrupalContentCrawler_** is the class where the data for indexing is resolved. The startCrawling function will do a GET request to all URLs saved in **drupalUrls** and prepare the next step of execution. All the content is saved in a map **topLevelJsonapiMap**. This process is running until **drupalUrls** has values. +If you need to check if the process is done you can check the value of **processFinished** flag. + +#### JSON:API +The content from Drupal URL has a JSON:API format. + +JSON:API is a specification for how a client should request the resources to be fetched or modified, and how a server should respond to those requests. +JSON:API is designed to minimize both the number of requests and the amount of data transmitted between clients and servers. This efficiency is achieved without compromising readability, flexibility, or discoverability. + +JSON:API requires use of the JSON:API media type `application/vnd.api+json` for exchanging data. + +## Dependencies +The most important dependency is the SDK connector. The SDK connector used in this project can be found [here](https://github.com/lucidworks/connectors-sdk-resources/tree/master/java-sdk). + +Beside this another needed dependency is a HTTP client in order to connect to a third-party REST API, in this project a Drupal Module. + +### OkHttp +OKHttp is a HTTP client that is efficient by default. It supports HTTP/2 and allows all requests to the same host to share a socket. It's connection pooling reduces request latency. The response caching avoids the network completely for repeat requests. +Using OkHttp is easy. Its request/response API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks. -## Connector properties +### Lombok +Lombok is a java library that automatically plugs into your editor and build tools and replaces using annotations most of the code regarding getters, setters and even constructors. -... \ No newline at end of file diff --git a/build.gradle b/build.gradle index 6d563ed..7fe0b05 100644 --- a/build.gradle +++ b/build.gradle @@ -1,4 +1,3 @@ - // Folder where the plugins are built ext.pluginsDir = "${rootProject.buildDir}" @@ -8,7 +7,10 @@ apply plugin: 'idea' group 'com.lucidworks.fusion.connector' version '1.0-SNAPSHOT' -sourceCompatibility = 1.8 +compileJava { + sourceCompatibility = 1.8 + targetCompatibility = 1.8 +} repositories { mavenCentral() @@ -16,6 +18,7 @@ repositories { maven { url "https://artifactory.lucidworks.com/artifactory/public-artifacts/" } + jcenter() } dependencies { @@ -28,6 +31,7 @@ dependencies { testCompile group: 'junit', name: 'junit', version: '4.12' compile group: 'org.slf4j', name: 'slf4j-api', version: '1.6.1' compile group: 'org.slf4j', name: 'slf4j-simple', version: '1.6.1' + testImplementation "org.mockito:mockito-core:${mockitoVersion}" } jar { diff --git a/diagram.png b/diagram.png new file mode 100644 index 0000000..77f79f7 Binary files /dev/null and b/diagram.png differ diff --git a/gradle.properties b/gradle.properties index aae4a3e..1691127 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,7 +1,7 @@ # packaing properties group=com.lucidworks.connector.plugins version=2.0.0 -# +# Connector SDK version connectorsSDKVersion=2.0.1 # Deploy tasks userPass=admin:a-very-secret-password @@ -16,7 +16,8 @@ lombokVersion=1.18.12 jacksonVersion=2.9.10 jsonApiVersion=0.10 okHttpVersion=4.7.2 -#plugins +mockitoVersion=2.7.22 +# plugins pluginClass=com.lucidworks.fusion.connector.ConnectorPlugin pluginId=com.lucidworks.fusion.connector pluginProvider=Lucidworks diff --git a/src/main/java/com/lucidworks/fusion/connector/Runner.java b/src/main/java/com/lucidworks/fusion/connector/Runner.java index 6077f7f..ac9c0b6 100644 --- a/src/main/java/com/lucidworks/fusion/connector/Runner.java +++ b/src/main/java/com/lucidworks/fusion/connector/Runner.java @@ -1,10 +1,13 @@ package com.lucidworks.fusion.connector; import com.fasterxml.jackson.databind.ObjectMapper; +import com.lucidworks.fusion.connector.model.DrupalLoginRequest; import com.lucidworks.fusion.connector.model.DrupalLoginResponse; +import com.lucidworks.fusion.connector.model.TopLevelJsonapi; import com.lucidworks.fusion.connector.service.ConnectorService; import com.lucidworks.fusion.connector.service.ContentService; import com.lucidworks.fusion.connector.service.DrupalOkHttp; +import com.lucidworks.fusion.connector.util.DataUtil; import java.util.Map; @@ -12,19 +15,36 @@ public class Runner { public static void main(String[] args) { String baseUrl = "http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/"; - DrupalOkHttp drupalOkHttp = new DrupalOkHttp(); ObjectMapper mapper = new ObjectMapper(); - ContentService contentService = new ContentService(drupalOkHttp, mapper); - DrupalLoginResponse drupalLoginResponse = contentService.login(baseUrl, "authenticated", "authenticated"); + DrupalOkHttp drupalOkHttp = new DrupalOkHttp(mapper); + ContentService contentService = new ContentService(mapper); - ConnectorService connectorService = new ConnectorService(baseUrl + "fusion", drupalLoginResponse, contentService); + DrupalLoginRequest drupalLoginRequest = new DrupalLoginRequest("authenticated", "authenticated"); + + DrupalLoginResponse drupalLoginResponse = drupalOkHttp.loginResponse(normalizeUrl(baseUrl) + normalizeUrl("/user/login"), drupalLoginRequest); + + ConnectorService connectorService = new ConnectorService(normalizeUrl(baseUrl) + normalizeUrl("/en/fusion/node/article"), new DrupalLoginResponse(), contentService, mapper); Map response = connectorService.prepareDataToUpload(); - response.forEach((currentUrl, content) -> { - System.out.println(currentUrl); - //System.out.println(content); - }); + Map topLevelJsonapiMap = contentService.getTopLevelJsonapiDataMap(); + + Map> objectMap = DataUtil.generateObjectMap(topLevelJsonapiMap); + + for (String key : objectMap.keySet()) { + Map pageContentMap = objectMap.get(key); + System.out.println(pageContentMap.values().toString()); + } + + System.out.println("Logout is successful: " + drupalOkHttp.logout(normalizeUrl(baseUrl) + "/user/logout", drupalLoginResponse)); + } + + + private static String normalizeUrl(String initialUrl) { + String normalizedUrl = initialUrl.endsWith("/") ? + initialUrl.substring(0, initialUrl.length() - 1) : initialUrl; + + return normalizedUrl; } } diff --git a/src/main/java/com/lucidworks/fusion/connector/config/ContentConfig.java b/src/main/java/com/lucidworks/fusion/connector/config/ContentConfig.java index 14ce057..d954ed7 100644 --- a/src/main/java/com/lucidworks/fusion/connector/config/ContentConfig.java +++ b/src/main/java/com/lucidworks/fusion/connector/config/ContentConfig.java @@ -29,7 +29,7 @@ interface Properties extends ConnectorPluginProperties { @Property( title = "Drupal URL", - description = "Content URL location. If empty, the connector will generate entries (see 'Generate Properties')", + description = "Page URL.", required = true, order = 1 ) @@ -38,8 +38,7 @@ interface Properties extends ConnectorPluginProperties { @Property( title = "Username for login", - description = "Username to login into drupal to be able to fetch content from it", - required = true, + description = "Username to login into drupal to be able to fetch content from it.", order = 2 ) @StringSchema @@ -47,13 +46,39 @@ interface Properties extends ConnectorPluginProperties { @Property( title = "Password for login", - description = "Password to login into drupal to be able to fetch content from it", - required = true, + description = "Password to login into drupal to be able to fetch content from it.", order = 3 ) - @StringSchema + @StringSchema() String getPassword(); + @Property( + title = "Login path", + description = "Login path.", + required = true, + order = 4 + ) + @StringSchema(defaultValue = "/user/login") + String getLoginPath(); + + @Property( + title = "Logout path", + description = "Logout path.", + required = true, + order = 5 + ) + @StringSchema(defaultValue = "/user/logout") + String getLogoutPath(); + + @Property( + title = "Drupal Content entry path", + description = "Drupal Content entry path from where the crawling should start.", + required = true, + order = 6 + ) + @StringSchema(defaultValue = "/en/fusion") + String getDrupalContentEntryPath(); + } } diff --git a/src/main/java/com/lucidworks/fusion/connector/fetcher/JsonContentFetcher.java b/src/main/java/com/lucidworks/fusion/connector/fetcher/JsonContentFetcher.java index 9618be5..da27839 100644 --- a/src/main/java/com/lucidworks/fusion/connector/fetcher/JsonContentFetcher.java +++ b/src/main/java/com/lucidworks/fusion/connector/fetcher/JsonContentFetcher.java @@ -1,19 +1,22 @@ package com.lucidworks.fusion.connector.fetcher; +import com.fasterxml.jackson.databind.ObjectMapper; import com.lucidworks.fusion.connector.config.ContentConfig; -import com.lucidworks.fusion.connector.content.DrupalContent; -import com.lucidworks.fusion.connector.content.DrupalContentEntry; import com.lucidworks.fusion.connector.exception.ServiceException; +import com.lucidworks.fusion.connector.model.DrupalLoginRequest; import com.lucidworks.fusion.connector.model.DrupalLoginResponse; +import com.lucidworks.fusion.connector.model.TopLevelJsonapi; import com.lucidworks.fusion.connector.plugin.api.fetcher.result.FetchResult; import com.lucidworks.fusion.connector.plugin.api.fetcher.type.content.ContentFetcher; -import com.lucidworks.fusion.connector.plugin.api.fetcher.type.content.FetchInput; import com.lucidworks.fusion.connector.service.ConnectorService; import com.lucidworks.fusion.connector.service.ContentService; +import com.lucidworks.fusion.connector.service.DrupalOkHttp; +import com.lucidworks.fusion.connector.util.DataUtil; import lombok.extern.slf4j.Slf4j; import javax.inject.Inject; import java.time.ZonedDateTime; +import java.util.HashMap; import java.util.Map; /** @@ -22,59 +25,36 @@ @Slf4j public class JsonContentFetcher implements ContentFetcher { - private final static String LAST_JOB_RUN_DATE_TIME = "lastJobRunDateTime"; - private final static String ENTRY_LAST_UPDATED = "lastUpdatedEntry"; - private final ContentConfig connectorConfig; private ContentService contentService; private ConnectorService connectorService; + private ObjectMapper objectMapper; + private DrupalOkHttp drupalOkHttp; + private DrupalLoginResponse drupalLoginResponse; @Inject public JsonContentFetcher( - ContentConfig connectorConfig, - ContentService contentService + ContentConfig connectorConfig ) { this.connectorConfig = connectorConfig; - this.contentService = contentService; - connectorService = new ConnectorService(getDrupalUrl(), null, contentService); - } - - private String getDrupalUrl() { - return connectorConfig.properties().getUrl(); - } - - private DrupalLoginResponse getDrupalLoginResponse() { - String username = connectorConfig.properties().getUsername(); - String password = connectorConfig.properties().getPassword(); - - DrupalLoginResponse drupalLoginResponse = contentService.login(getDrupalUrl(), username, password); - - return drupalLoginResponse; - + this.objectMapper = new ObjectMapper(); + this.drupalOkHttp = new DrupalOkHttp(objectMapper); + this.contentService = new ContentService(objectMapper); + this.drupalLoginResponse = getDrupalLoginResponse(); + this.connectorService = new ConnectorService(getDrupalContentEntryUrl(), this.drupalLoginResponse, contentService, objectMapper); } @Override public FetchResult fetch(FetchContext fetchContext) { - try { - FetchInput input = fetchContext.getFetchInput(); - Map metaData = input.getMetadata(); - Map contentMap = connectorService.prepareDataToUpload(); + Map topLevelJsonapiMap = new HashMap<>(); + Map contentMap = new HashMap<>(); - contentMap.forEach((url, content) -> { + try { + contentMap = connectorService.prepareDataToUpload(); - fetchContext.newCandidate(url) - .metadata(m -> { - m.setString("content", content); - }).emit(); + topLevelJsonapiMap = contentService.getTopLevelJsonapiDataMap(); - fetchContext.newDocument(input.getId()) - .fields(f -> { - f.setString("content_s", (String) metaData.get("content")); - f.setLong("lastUpdatedEntry_l", ZonedDateTime.now().toEpochSecond()); - }) - .emit(); - }); } catch (ServiceException e) { String message = "Failed to parse content from Drupal!"; log.error(message, e); @@ -83,21 +63,74 @@ public FetchResult fetch(FetchContext fetchContext) { .emit(); } + if (contentMap.keySet().size() == topLevelJsonapiMap.keySet().size()) { + + Map> objectMap = DataUtil.generateObjectMap(topLevelJsonapiMap); + + for (String key : objectMap.keySet()) { + Map pageContentMap = objectMap.get(key); + fetchContext.newDocument(key) + .fields(field -> { + field.setString("url", key); + field.setLong("lastUpdated", ZonedDateTime.now().toEpochSecond()); + field.merge(pageContentMap); + }) + .emit(); + } + } else { + String message = "Failed to store all Drupal Content."; + log.error(message); + fetchContext.newError(fetchContext.getFetchInput().getId()) + .withError(message) + .emit(); + } + + logout(); + return fetchContext.newResult(); } - private void emitDrupalCandidates(DrupalContent feed, FetchContext fetchContext, long lastJobRunDateTime) { - Map entryMap = feed.getEntries(); - entryMap.forEach((id, entry) -> { - fetchContext.newCandidate(id) - .metadata(m -> { - m.setString("content", entry.getContent()); - // add last time when entry was modified - m.setLong(ENTRY_LAST_UPDATED, entry.getLastUpdated()); - // add 'lastJobRunDateTime'. - m.setLong(LAST_JOB_RUN_DATE_TIME, lastJobRunDateTime); - }) - .emit(); - }); + private DrupalLoginResponse getDrupalLoginResponse() { + String username = connectorConfig.properties().getUsername(); + String password = connectorConfig.properties().getPassword(); + + + if (username != null && !username.isEmpty() && + password != null && !password.isEmpty()) { + DrupalLoginRequest drupalLoginRequest = new DrupalLoginRequest(username, password); + + drupalLoginResponse = drupalOkHttp.loginResponse(getDrupalLoginUrl(), drupalLoginRequest); + + return drupalLoginResponse; + } else { + return new DrupalLoginResponse(); + } + } + + private String normalizeUrl(String initialUrl) { + String normalizedUrl = initialUrl.endsWith("/") ? + initialUrl.substring(0, initialUrl.length() - 1) : initialUrl; + + return normalizedUrl; + } + + private boolean logout() { + return drupalOkHttp.logout(getDrupalLogoutUrl(), drupalLoginResponse); + } + + private String getDrupalUrl() { + return connectorConfig.properties().getUrl(); + } + + private String getDrupalContentEntryUrl() { + return normalizeUrl(getDrupalUrl()) + normalizeUrl(connectorConfig.properties().getDrupalContentEntryPath()); + } + + private String getDrupalLoginUrl() { + return normalizeUrl(getDrupalUrl()) + normalizeUrl(connectorConfig.properties().getLoginPath()); + } + + private String getDrupalLogoutUrl() { + return normalizeUrl(getDrupalUrl()) + normalizeUrl(connectorConfig.properties().getLogoutPath()); } } diff --git a/src/main/java/com/lucidworks/fusion/connector/model/AttributeBody.java b/src/main/java/com/lucidworks/fusion/connector/model/AttributeBody.java new file mode 100644 index 0000000..9308439 --- /dev/null +++ b/src/main/java/com/lucidworks/fusion/connector/model/AttributeBody.java @@ -0,0 +1,15 @@ +package com.lucidworks.fusion.connector.model; + +import lombok.Getter; +import lombok.Setter; +import lombok.ToString; + +@Getter +@Setter +@ToString +public class AttributeBody { + private String value; + private String format; + private String processed; + private String summary; +} diff --git a/src/main/java/com/lucidworks/fusion/connector/model/Attributes.java b/src/main/java/com/lucidworks/fusion/connector/model/Attributes.java index 1a6a977..b379d0a 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/Attributes.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/Attributes.java @@ -16,7 +16,7 @@ public class Attributes implements Serializable { @JsonProperty("drupal_internal__nid") - private Integer drupalInternalTid; + private Integer drupalInternalNid; @JsonProperty("drupal_internal__vid") private Integer drupalInternalVid; @@ -38,6 +38,11 @@ public class Attributes implements Serializable { private boolean revisionTranslationAffected; private Path path; + private String title; + private AttributeBody body; + + private String name; + private Uri uri; private Map fields = new HashMap<>(); diff --git a/src/main/java/com/lucidworks/fusion/connector/model/Data.java b/src/main/java/com/lucidworks/fusion/connector/model/Data.java index 65e61de..fb0d882 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/Data.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/Data.java @@ -1,7 +1,5 @@ package com.lucidworks.fusion.connector.model; -import com.github.jasminb.jsonapi.annotations.Id; -import com.github.jasminb.jsonapi.annotations.Type; import lombok.Getter; import lombok.Setter; import lombok.ToString; @@ -12,9 +10,7 @@ @Getter @Setter @ToString -@Type("data") public class Data implements Serializable { - @Id private String id; private String type; private Map Links; diff --git a/src/main/java/com/lucidworks/fusion/connector/model/DrupalLoginResponse.java b/src/main/java/com/lucidworks/fusion/connector/model/DrupalLoginResponse.java index bff85ed..2365920 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/DrupalLoginResponse.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/DrupalLoginResponse.java @@ -1,5 +1,6 @@ package com.lucidworks.fusion.connector.model; +import com.fasterxml.jackson.annotation.JsonIgnore; import com.fasterxml.jackson.annotation.JsonProperty; import lombok.Getter; import lombok.Setter; @@ -16,8 +17,6 @@ public class DrupalLoginResponse implements Serializable { private String csrfToken; @JsonProperty("logout_token") private String logoutToken; - - public String getAuthorization() { - return new StringBuilder().append("Bearer " + csrfToken).toString(); - } + @JsonIgnore + private String cookie; } diff --git a/src/main/java/com/lucidworks/fusion/connector/model/Errors.java b/src/main/java/com/lucidworks/fusion/connector/model/Errors.java index 18f0860..2344816 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/Errors.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/Errors.java @@ -1,6 +1,5 @@ package com.lucidworks.fusion.connector.model; -import com.github.jasminb.jsonapi.annotations.Type; import lombok.Getter; import lombok.Setter; @@ -8,7 +7,6 @@ @Getter @Setter -@Type("errors") public class Errors implements Serializable { private String title; private String status; diff --git a/src/main/java/com/lucidworks/fusion/connector/model/Jsonapi.java b/src/main/java/com/lucidworks/fusion/connector/model/Jsonapi.java index e68400d..1c96798 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/Jsonapi.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/Jsonapi.java @@ -1,6 +1,5 @@ package com.lucidworks.fusion.connector.model; -import com.github.jasminb.jsonapi.annotations.Type; import lombok.Getter; import lombok.Setter; import lombok.ToString; @@ -10,10 +9,7 @@ @Getter @Setter @ToString -@Type("jsonApi") public class Jsonapi implements Serializable { private String version; - - @com.github.jasminb.jsonapi.annotations.Meta private Meta meta; } diff --git a/src/main/java/com/lucidworks/fusion/connector/model/Meta.java b/src/main/java/com/lucidworks/fusion/connector/model/Meta.java index 7ffe705..b181d39 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/Meta.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/Meta.java @@ -1,6 +1,5 @@ package com.lucidworks.fusion.connector.model; -import com.github.jasminb.jsonapi.annotations.Type; import lombok.Getter; import lombok.Setter; import lombok.ToString; @@ -11,7 +10,6 @@ @Getter @Setter @ToString -@Type("meta") public class Meta implements Serializable { private Map Links; private Object omitted; diff --git a/src/main/java/com/lucidworks/fusion/connector/model/RelationshipFields.java b/src/main/java/com/lucidworks/fusion/connector/model/RelationshipFields.java index b42d924..9262aae 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/RelationshipFields.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/RelationshipFields.java @@ -10,8 +10,6 @@ @Setter @ToString public class RelationshipFields { - - // TODO - set Data and Data[] for the same json field private Object data; private Map Links; } diff --git a/src/main/java/com/lucidworks/fusion/connector/model/TopLevelJsonapi.java b/src/main/java/com/lucidworks/fusion/connector/model/TopLevelJsonapi.java index 9aa0398..ef62e7c 100644 --- a/src/main/java/com/lucidworks/fusion/connector/model/TopLevelJsonapi.java +++ b/src/main/java/com/lucidworks/fusion/connector/model/TopLevelJsonapi.java @@ -1,6 +1,5 @@ package com.lucidworks.fusion.connector.model; -import com.github.jasminb.jsonapi.annotations.Type; import lombok.Getter; import lombok.Setter; import lombok.ToString; @@ -11,7 +10,6 @@ @Getter @Setter @ToString -@Type("topLevel") public class TopLevelJsonapi implements Serializable { private Jsonapi jsonapi; private Data[] data; diff --git a/src/main/java/com/lucidworks/fusion/connector/model/Uri.java b/src/main/java/com/lucidworks/fusion/connector/model/Uri.java new file mode 100644 index 0000000..924e6d4 --- /dev/null +++ b/src/main/java/com/lucidworks/fusion/connector/model/Uri.java @@ -0,0 +1,13 @@ +package com.lucidworks.fusion.connector.model; + +import lombok.Getter; +import lombok.Setter; +import lombok.ToString; + +@Getter +@Setter +@ToString +public class Uri { + private String value; + private String url; +} diff --git a/src/main/java/com/lucidworks/fusion/connector/service/ConnectorService.java b/src/main/java/com/lucidworks/fusion/connector/service/ConnectorService.java index 5d697ed..4c1d24a 100644 --- a/src/main/java/com/lucidworks/fusion/connector/service/ConnectorService.java +++ b/src/main/java/com/lucidworks/fusion/connector/service/ConnectorService.java @@ -1,5 +1,6 @@ package com.lucidworks.fusion.connector.service; +import com.fasterxml.jackson.databind.ObjectMapper; import com.lucidworks.fusion.connector.exception.ServiceException; import com.lucidworks.fusion.connector.model.DrupalLoginResponse; import lombok.extern.slf4j.Slf4j; @@ -15,8 +16,8 @@ public class ConnectorService { private final DrupalContentCrawler drupalContentCrawler; private boolean isProcessStarted = false; - public ConnectorService(String drupalUrl, DrupalLoginResponse drupalLoginResponse, ContentService contentService) { - this.drupalContentCrawler = new DrupalContentCrawler(drupalUrl, drupalLoginResponse, contentService); + public ConnectorService(String drupalUrl, DrupalLoginResponse drupalLoginResponse, ContentService contentService, ObjectMapper mapper) { + this.drupalContentCrawler = new DrupalContentCrawler(drupalUrl, drupalLoginResponse, contentService, mapper); } /** diff --git a/src/main/java/com/lucidworks/fusion/connector/service/ContentService.java b/src/main/java/com/lucidworks/fusion/connector/service/ContentService.java index 6e7bf4a..399c565 100644 --- a/src/main/java/com/lucidworks/fusion/connector/service/ContentService.java +++ b/src/main/java/com/lucidworks/fusion/connector/service/ContentService.java @@ -2,23 +2,17 @@ import com.fasterxml.jackson.databind.ObjectMapper; import com.lucidworks.fusion.connector.exception.ServiceException; -import com.lucidworks.fusion.connector.model.Data; -import com.lucidworks.fusion.connector.model.DrupalLoginRequest; -import com.lucidworks.fusion.connector.model.DrupalLoginResponse; -import com.lucidworks.fusion.connector.model.RelationshipFields; -import com.lucidworks.fusion.connector.model.TopLevelJsonApiData; -import com.lucidworks.fusion.connector.model.TopLevelJsonapi; +import com.lucidworks.fusion.connector.model.*; import lombok.extern.slf4j.Slf4j; -import okhttp3.ResponseBody; import javax.inject.Inject; import java.io.IOException; +import java.util.Map; +import java.util.HashMap; +import java.util.List; import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; -import java.util.HashMap; -import java.util.List; -import java.util.Map; /** * Content Service fetch the content from Drupal @@ -26,17 +20,15 @@ @Slf4j public class ContentService { - private final String SELF_LINK = "self"; + private static final String SELF_LINK = "self"; + private static final String HTML_LINK = "html"; - private final DrupalOkHttp drupalOkHttp; private final ObjectMapper mapper; private Map topLevelJsonapiDataMap; @Inject - public ContentService(DrupalOkHttp drupalOkHttp, ObjectMapper objectMapper) { - this.drupalOkHttp = drupalOkHttp; + public ContentService(ObjectMapper objectMapper) { this.mapper = objectMapper; - topLevelJsonapiDataMap = new HashMap<>(); } @@ -50,7 +42,7 @@ public List collectLinksFromDrupalContent(String url, String content) { log.info("Enter collectLinksFromDrupalContent method..."); List links = new ArrayList<>(); - TopLevelJsonapi topLevelJsonapi = null; + TopLevelJsonapi topLevelJsonapi; try { topLevelJsonapi = mapper.readValue(content, TopLevelJsonapi.class); @@ -74,7 +66,7 @@ public List collectLinksFromDrupalContent(String url, String content) { Collection relationshipFields = data.getRelationships().getFields().values(); relationshipFields.forEach(fields -> { fields.getLinks().forEach((linkTag, linkHref) -> { - if (!linkTag.equals(SELF_LINK)) { + if (!linkTag.equals(SELF_LINK) && !linkTag.equals(HTML_LINK)) { links.add(linkHref.getHref()); } }); @@ -83,6 +75,7 @@ public List collectLinksFromDrupalContent(String url, String content) { } if (topLevelJsonapi.getLinks() != null || !topLevelJsonapi.getLinks().isEmpty()) { + topLevelJsonapiDataMap.put(url, topLevelJsonapi); topLevelJsonapi.getLinks().forEach((linkTag, linkHref) -> { if (!linkTag.equals(SELF_LINK)) { links.add(linkHref.getHref()); @@ -93,30 +86,7 @@ public List collectLinksFromDrupalContent(String url, String content) { return links; } - /** - * Request to login the user in order to have the JWT token - * - * @param url - * @param username - * @param password - * @return - */ - public DrupalLoginResponse login(String url, String username, String password) { - log.info("Trying to login the user {}", username); - - DrupalLoginRequest drupalLoginRequest = new DrupalLoginRequest(username, password); - - ResponseBody loginResponse = drupalOkHttp.login(url, drupalLoginRequest); - - DrupalLoginResponse drupalLoginResponse = null; - try { - drupalLoginResponse = mapper.readValue(loginResponse.string(), DrupalLoginResponse.class); - } catch (IOException e) { - throw new ServiceException("Failed to get the loginResponse from login request.", e); - } - - log.info("User: {} logged in", username); - return drupalLoginResponse; + public Map getTopLevelJsonapiDataMap(){ + return topLevelJsonapiDataMap; } - } diff --git a/src/main/java/com/lucidworks/fusion/connector/service/DrupalContentCrawler.java b/src/main/java/com/lucidworks/fusion/connector/service/DrupalContentCrawler.java index 6321e58..f795473 100644 --- a/src/main/java/com/lucidworks/fusion/connector/service/DrupalContentCrawler.java +++ b/src/main/java/com/lucidworks/fusion/connector/service/DrupalContentCrawler.java @@ -1,14 +1,16 @@ package com.lucidworks.fusion.connector.service; +import com.fasterxml.jackson.databind.ObjectMapper; import com.lucidworks.fusion.connector.exception.ServiceException; import com.lucidworks.fusion.connector.model.DrupalLoginResponse; +import com.lucidworks.fusion.connector.model.TopLevelJsonapi; import lombok.extern.slf4j.Slf4j; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Arrays; /** * Drupal Content Crawler can create a Map with all links and content from them. @@ -22,6 +24,7 @@ public class DrupalContentCrawler { private DrupalLoginResponse loggedInUser; private DrupalOkHttp drupalOkHttp; private ContentService contentService; + private Map topLevelJsonapiMap; /** * Constructor for Crawler @@ -30,12 +33,14 @@ public class DrupalContentCrawler { * @param loggedInUser the loggedInUser with JWT token inside * @param contentService the content service class */ - public DrupalContentCrawler(String drupalUrl, DrupalLoginResponse loggedInUser, ContentService contentService) { - this.drupalOkHttp = new DrupalOkHttp(); + public DrupalContentCrawler(String drupalUrl, DrupalLoginResponse loggedInUser, ContentService contentService, + ObjectMapper mapper) { + this.drupalOkHttp = new DrupalOkHttp(mapper); this.loggedInUser = loggedInUser; this.drupalUrls = new ArrayList<>(Arrays.asList(drupalUrl)); this.visitedUrls = new HashMap<>(); + this.topLevelJsonapiMap = new HashMap<>(); this.contentService = contentService; } @@ -47,6 +52,7 @@ public DrupalContentCrawler(String drupalUrl, DrupalLoginResponse loggedInUser, public void startCrawling() { log.info("Enter startCrawling method."); + processFinished = false; Map currentStepContent = new HashMap<>(); List urlsVisitedInCurrentStep = new ArrayList<>(); try { @@ -64,8 +70,11 @@ public void startCrawling() { currentStepContent.forEach((url, content) -> { drupalUrls.addAll(contentService.collectLinksFromDrupalContent(url, content)); visitedUrls.put(url, content); + topLevelJsonapiMap.putAll(contentService.getTopLevelJsonapiDataMap()); }); + drupalUrls.removeIf(drupalUrl -> visitedUrls.containsKey(drupalUrl)); + urlsVisitedInCurrentStep.clear(); currentStepContent.clear(); diff --git a/src/main/java/com/lucidworks/fusion/connector/service/DrupalOkHttp.java b/src/main/java/com/lucidworks/fusion/connector/service/DrupalOkHttp.java index 4e1523e..3cba3a9 100644 --- a/src/main/java/com/lucidworks/fusion/connector/service/DrupalOkHttp.java +++ b/src/main/java/com/lucidworks/fusion/connector/service/DrupalOkHttp.java @@ -1,26 +1,32 @@ package com.lucidworks.fusion.connector.service; +import com.fasterxml.jackson.databind.ObjectMapper; import com.lucidworks.fusion.connector.exception.RequestException; +import com.lucidworks.fusion.connector.exception.ServiceException; import com.lucidworks.fusion.connector.model.DrupalLoginRequest; import com.lucidworks.fusion.connector.model.DrupalLoginResponse; import lombok.extern.slf4j.Slf4j; +import okhttp3.RequestBody; import okhttp3.MediaType; import okhttp3.OkHttpClient; import okhttp3.Request; -import okhttp3.RequestBody; import okhttp3.Response; -import okhttp3.ResponseBody; import java.io.IOException; @Slf4j public class DrupalOkHttp { - public static final MediaType JSON - = MediaType.parse("application/json; charset=utf-8"); - private final OkHttpClient okHttpClient = new OkHttpClient(); + private static final MediaType JSON = MediaType.parse("application/json; charset=utf-8"); + private static final String FORMAT = "?_format=json"; + private static final String CRSF_TOKEN = "&crsf_token="; + + private OkHttpClient okHttpClient; + private ObjectMapper mapper; - public DrupalOkHttp() { + public DrupalOkHttp(ObjectMapper objectMapper) { + okHttpClient = new OkHttpClient.Builder().build(); + this.mapper = objectMapper; } /** @@ -31,10 +37,11 @@ public DrupalOkHttp() { * @return */ public String getDrupalContent(String url, DrupalLoginResponse drupalLoginResponse) { + String cookies = drupalLoginResponse.getCookie() != null ? drupalLoginResponse.getCookie() : ""; Request getRequest = new Request.Builder() .url(url) .addHeader("Content-Type", "application/vnd.api+json") - //.addHeader("Authorization", drupalLoginResponse.getAuthorization()) + .addHeader("Cookies", cookies) .build(); try { @@ -62,20 +69,59 @@ public String getDrupalContent(String url, DrupalLoginResponse drupalLoginRespon * @param drupalLoginRequest Object that contains the user's credentials * @return */ - public ResponseBody login(String url, DrupalLoginRequest drupalLoginRequest) { - RequestBody requestBody = RequestBody.create(drupalLoginRequest.getJson(), JSON); + public DrupalLoginResponse loginResponse(String url, DrupalLoginRequest drupalLoginRequest) { + String loginUrl = url + FORMAT; + RequestBody requestBody = + RequestBody.Companion.create(drupalLoginRequest.getJson(), JSON); Request loginRequest = new Request.Builder() - .url(url + "user/login?_format=json") + .url(loginUrl) .post(requestBody) .build(); + String loginResponse, cookie; try { Response response = okHttpClient.newCall(loginRequest).execute(); - return response.body(); + loginResponse = response.body().string(); + cookie = response.headers().get("Set-Cookie").split(";")[0]; } catch (IOException exception) { throw new RequestException("There was an error when trying to login the user: " + drupalLoginRequest.getName(), exception); } + + DrupalLoginResponse drupalLoginResponse; + try { + drupalLoginResponse = mapper.readValue(loginResponse, DrupalLoginResponse.class); + } catch (IOException e) { + throw new ServiceException("Failed to get the loginResponse from login request.", e); + } + drupalLoginResponse.setCookie(cookie); + + log.info("User: {} logged in", drupalLoginRequest.getName()); + return drupalLoginResponse; + } + + /** + * Logout function + * + * @param url + * @param drupalLoginResponse + * @return true if the code from logout request is 200 or 403 + */ + public boolean logout(String url, DrupalLoginResponse drupalLoginResponse) { + String logoutUrl = url + FORMAT + CRSF_TOKEN + drupalLoginResponse.getCsrfToken(); + + Request logoutRequest = new Request.Builder() + .url(logoutUrl) + .addHeader("Cookies", drupalLoginResponse.getCookie()) + .build(); + + try { + Response response = okHttpClient.newCall(logoutRequest).execute(); + //200 OK or 403 Forbidden is success + return response.code() == 200 || response.code() == 403; + } catch (IOException e) { + throw new ServiceException("Failed to logout", e); + } } } diff --git a/src/main/java/com/lucidworks/fusion/connector/util/DataUtil.java b/src/main/java/com/lucidworks/fusion/connector/util/DataUtil.java new file mode 100644 index 0000000..0ebf339 --- /dev/null +++ b/src/main/java/com/lucidworks/fusion/connector/util/DataUtil.java @@ -0,0 +1,310 @@ +package com.lucidworks.fusion.connector.util; + +import com.lucidworks.fusion.connector.model.Data; +import com.lucidworks.fusion.connector.model.TopLevelJsonapi; + +import java.util.Map; +import java.util.HashMap; +import java.util.ArrayList; +import java.util.List; + +/** + * Util class to extract the content of every field from a Drupal page. + */ +public final class DataUtil { + + private DataUtil() { + } + + /** + * Generate all the content from every page and add it to a map + * + * @param topLevelJsonapiMap + * @return The map with all the content indexed + */ + public static Map> generateObjectMap(Map topLevelJsonapiMap) { + Map> allObjectsMap = new HashMap<>(); + + for (String url : topLevelJsonapiMap.keySet()) { + TopLevelJsonapi topLevelJsonapi = topLevelJsonapiMap.get(url); + + if (topLevelJsonapi.getData() != null) { + for (Data data : topLevelJsonapi.getData()) { + Map dataMap = prepareDataMap(data); + dataMap = prepareRelationshipFields(dataMap, topLevelJsonapiMap, data); + if (dataMap.get("html.link") != null) { + allObjectsMap.put(dataMap.get("html.link").toString(), dataMap); + } + } + } + } + + return allObjectsMap; + } + + private static Map prepareRelationshipFields(Map dataMap, Map topLevelJsonapiMap, Data currentData) { + + String tagsUrl = getRelatedUrl(currentData, "tags"); + String categoryUrl = getRelatedUrl(currentData, "category"); + String imageUrl = getRelatedUrl(currentData, "image"); + + if (!tagsUrl.isEmpty() && topLevelJsonapiMap.get(tagsUrl) != null && topLevelJsonapiMap.get(tagsUrl).getData() != null) { + + List tags = new ArrayList<>(); + + for (Data data : topLevelJsonapiMap.get(tagsUrl).getData()) { + tags.add(data.getAttributes().getName()); + } + + if (!tags.isEmpty()) { + dataMap.put("tags", tags); + } + } + + if (!categoryUrl.isEmpty() && topLevelJsonapiMap.get(categoryUrl) != null && topLevelJsonapiMap.get(categoryUrl).getData() != null) { + List category = new ArrayList<>(); + + for (Data data : topLevelJsonapiMap.get(categoryUrl).getData()) { + category.add(data.getAttributes().getName()); + } + + if (!category.isEmpty()) { + dataMap.put("category", category); + } + } + + if (!imageUrl.isEmpty() && topLevelJsonapiMap.get(imageUrl) != null && topLevelJsonapiMap.get(imageUrl).getData() != null) { + List images = new ArrayList<>(); + + for (Data data : topLevelJsonapiMap.get(imageUrl).getData()) { + + String image2Url = getRelatedUrl(data, "image"); + + if (image2Url != null && topLevelJsonapiMap.get(image2Url) != null && topLevelJsonapiMap.get(image2Url).getData() != null) { + Data[] currentDataList = topLevelJsonapiMap.get(image2Url).getData(); + for (Data currentStepData : currentDataList) { + if (currentStepData.getAttributes().getUri() != null) { + images.add(currentStepData.getAttributes().getUri().getUrl()); + } + } + } else if (data.getAttributes().getUri()!= null){ + images.add(data.getAttributes().getUri().getUrl()); + } + } + + if (!images.isEmpty()) { + dataMap.put("image", images); + } + } + + return dataMap; + } + + private static String getRelatedUrl(Data data, String requiredValue) { + String url = ""; + + if (data.getRelationships() != null) { + for (String keySet : data.getRelationships().getFields().keySet()) { + if (keySet.contains(requiredValue)) { + if (data.getRelationships().getFields().get(keySet).getLinks().get("related") != null) { + url = data.getRelationships().getFields().get(keySet).getLinks().get("related").getHref(); + } + } + } + } + + return url; + } + + private static Map prepareDataMap(Data data) { + Map dataMap = new HashMap<>(); + + if (!DataUtil.getDataId(data).isEmpty()) { + dataMap.put("id", DataUtil.getDataId(data)); + } + + if (!DataUtil.getDataType(data).isEmpty()) { + dataMap.put("type", DataUtil.getDataType(data)); + } + + if (!DataUtil.getDataLinks(data).isEmpty()) { + dataMap.put("links", DataUtil.getDataLinks(data)); + } + + if (!DataUtil.getDataAttributeDrupalInternalNid(data).isEmpty()) { + dataMap.put("drupal.internal.nid", DataUtil.getDataAttributeDrupalInternalNid(data)); + } + + if (!DataUtil.getDataAttributeDrupalInternalVid(data).isEmpty()) { + dataMap.put("drupal.internal.vid", DataUtil.getDataAttributeDrupalInternalVid(data)); + } + + if (!DataUtil.getDataAttributeLangcode(data).isEmpty()) { + dataMap.put("langcode", DataUtil.getDataAttributeLangcode(data)); + } + + if (!DataUtil.getDataAttributeRevisionTimestamp(data).isEmpty()) { + dataMap.put("revision.timestamp", DataUtil.getDataAttributeRevisionTimestamp(data)); + } + + if (!DataUtil.getDataAttributeRevisionLog(data).isEmpty()) { + dataMap.put("revision.log", DataUtil.getDataAttributeRevisionLog(data)); + } + + dataMap.put("status", DataUtil.getDataAttributeStatus(data)); + + if (!DataUtil.getDataAttributeChanged(data).isEmpty()) { + dataMap.put("changed", DataUtil.getDataAttributeChanged(data)); + } + + dataMap.put("defaultLangcode", DataUtil.getDataAttributeDefaultLangcode(data)); + dataMap.put("revision.translation.affected", DataUtil.getDataAttributeRevisionTranslationAffected(data)); + + if (!DataUtil.getDataAttributeTitle(data).isEmpty()) { + dataMap.put("title", DataUtil.getDataAttributeTitle(data)); + } + + if (!DataUtil.getDataAttributePath(data).isEmpty()) { + dataMap.put("path", DataUtil.getDataAttributePath(data)); + } + + if (data.getAttributes().getBody() != null) { + dataMap.putAll(DataUtil.getDataAttributeBody(data)); + } + + if (!data.getAttributes().getFields().isEmpty()) { + dataMap.putAll(DataUtil.getDataAttributeFields(data)); + } + + if (!data.getRelationships().getFields().isEmpty()) { + dataMap.putAll(DataUtil.getDataRelationships(data)); + } + + if (!DataUtil.getDataHtmlLink(data).isEmpty()) { + dataMap.put("html.link", DataUtil.getDataHtmlLink(data)); + } + + return dataMap; + } + + private static String getDataId(Data data) { + return data.getId() != null ? data.getId() : ""; + } + + private static String getDataType(Data data) { + return data.getType() != null ? data.getType() : ""; + } + + private static String getDataLinks(Data data) { + return data.getLinks() != null ? data.getLinks().toString() : ""; + } + + private static String getDataAttributeDrupalInternalNid(Data data) { + return data.getAttributes().getDrupalInternalNid() != null ? String.valueOf(data.getAttributes().getDrupalInternalNid()) : ""; + } + + private static String getDataAttributeDrupalInternalVid(Data data) { + return data.getAttributes().getDrupalInternalVid() != null ? String.valueOf(data.getAttributes().getDrupalInternalVid()) : ""; + } + + private static String getDataAttributeLangcode(Data data) { + return data.getAttributes().getLangcode() != null ? data.getAttributes().getLangcode() : ""; + } + + private static String getDataAttributeRevisionTimestamp(Data data) { + return data.getAttributes().getRevisionCreated() != null ? data.getAttributes().getRevisionCreated() : ""; + } + + private static String getDataAttributeRevisionLog(Data data) { + return data.getAttributes().getRevisionLogMessage() != null ? data.getAttributes().getRevisionLogMessage() : ""; + } + + private static boolean getDataAttributeStatus(Data data) { + return data.getAttributes().isStatus(); + } + + private static String getDataAttributeChanged(Data data) { + return data.getAttributes().getChanged() != null ? data.getAttributes().getChanged() : ""; + } + + private static boolean getDataAttributeDefaultLangcode(Data data) { + return data.getAttributes().isDefaultLangcode(); + } + + private static boolean getDataAttributeRevisionTranslationAffected(Data data) { + return data.getAttributes().isRevisionTranslationAffected(); + } + + private static String getDataAttributeTitle(Data data) { + return data.getAttributes().getTitle() != null ? data.getAttributes().getTitle() : ""; + } + + private static String getDataAttributePath(Data data) { + return data.getAttributes().getPath() != null ? data.getAttributes().getPath().toString() : ""; + } + + private static Map getDataAttributeBody(Data data) { + Map bodyFields = new HashMap<>(); + + if (data.getAttributes().getBody().getValue() != null) { + bodyFields.put("body.value", data.getAttributes().getBody().getValue()); + } + + if (data.getAttributes().getBody().getFormat() != null) { + bodyFields.put("body.format", data.getAttributes().getBody().getFormat()); + } + + if (data.getAttributes().getBody().getProcessed() != null) { + bodyFields.put("body.processed", data.getAttributes().getBody().getProcessed()); + } + + if (data.getAttributes().getBody().getSummary() != null) { + bodyFields.put("body.summary", data.getAttributes().getBody().getSummary()); + } + + return bodyFields; + } + + private static Map getDataAttributeFields(Data data) { + Map fieldsMap = new HashMap<>(); + + if (data.getAttributes() != null && data.getAttributes().getFields() != null) { + for (String key : data.getAttributes().getFields().keySet()) { + if (data.getAttributes().getFields().get(key) != null) { + Object fieldObject = data.getAttributes().getFields().get(key); + if (fieldObject instanceof Map) { + Map fieldObjectMap = Map.class.cast(fieldObject); + for (String fieldObjectKey : fieldObjectMap.keySet()) { + fieldsMap.put(key + "_" + fieldObjectKey, fieldObjectMap.get(fieldObjectKey)); + } + } else { + fieldsMap.put(key, data.getAttributes().getFields().get(key)); + } + } + } + } + + + return fieldsMap; + } + + private static Map getDataRelationships(Data data) { + Map fieldsMap = new HashMap<>(); + + for (String key : data.getRelationships().getFields().keySet()) { + fieldsMap.put(key, data.getRelationships().getFields().get(key)); + } + + return fieldsMap; + } + + private static String getDataHtmlLink(Data data) { + String htmlLink = ""; + + if (data.getLinks() != null && data.getLinks().size() > 0) { + htmlLink = data.getLinks().get("html") != null ? data.getLinks().get("html").getHref() : ""; + } + + return htmlLink; + } +} diff --git a/src/test/java/com/lucidworks/fusion/connector/service/ConnectorServiceTest.java b/src/test/java/com/lucidworks/fusion/connector/service/ConnectorServiceTest.java new file mode 100644 index 0000000..3fc74d6 --- /dev/null +++ b/src/test/java/com/lucidworks/fusion/connector/service/ConnectorServiceTest.java @@ -0,0 +1,75 @@ +package com.lucidworks.fusion.connector.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.lucidworks.fusion.connector.model.DrupalLoginResponse; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.MockitoAnnotations; +import org.mockito.MockitoSession; +import org.mockito.quality.Strictness; + +import java.util.Map; + +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.when; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doThrow; + +public class ConnectorServiceTest { + + private static final String URL = "http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion"; + + @Mock + private ConnectorService connectorService; + + @Mock + private DrupalContentCrawler drupalContentCrawler; + + @Mock + private ContentService contentService; + + @Mock + private DrupalLoginResponse drupalLoginResponse; + + @Mock + private ObjectMapper mapper; + + private MockitoSession mockitoSession; + + @Before + public void setUp() { + + mockitoSession = Mockito.mockitoSession() + .initMocks(this) + .strictness(Strictness.LENIENT) + .startMocking(); + MockitoAnnotations.initMocks(this); + + connectorService = new ConnectorService(URL, drupalLoginResponse, contentService, mapper); + } + + @After + public void tearDown() { + mockitoSession.finishMocking(); + } + + @Test + public void testPrepareDataToUpload() { + when(drupalContentCrawler.isProcessFinished()).thenReturn(false); + doNothing().when(drupalContentCrawler).startCrawling(); + Map dataMap = connectorService.prepareDataToUpload(); + + assertEquals(1, dataMap.size()); + assertEquals(true, dataMap.containsKey(URL)); + + } + + @Test(expected = RuntimeException.class) + public void testPrepareDataToUpload_ThrowException() { + doThrow().when(drupalContentCrawler).startCrawling(); + } + +} diff --git a/src/test/java/com/lucidworks/fusion/connector/service/ContentServiceTest.java b/src/test/java/com/lucidworks/fusion/connector/service/ContentServiceTest.java index 8c64ae8..ab124b0 100644 --- a/src/test/java/com/lucidworks/fusion/connector/service/ContentServiceTest.java +++ b/src/test/java/com/lucidworks/fusion/connector/service/ContentServiceTest.java @@ -1,4 +1,88 @@ package com.lucidworks.fusion.connector.service; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.junit.Assert.assertEquals; + public class ContentServiceTest { + + private static final int TOTAL = 5; + + private ContentService contentService; + private ObjectMapper mapper; + + @Before + public void setUp() { + mapper = new ObjectMapper(); + contentService = new ContentService(mapper); + } + + @Test + public void testCollectLinksFromDrupalContent() { + String url = "http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion"; + String content = prepareContent(); + List expectedLinks = prepareListWithExpectedLinks(); + List links = contentService.collectLinksFromDrupalContent(url, content); + + assertEquals(TOTAL, links.size()); + assertEquals(expectedLinks.get(0), links.get(0)); + assertEquals(expectedLinks.get(1), links.get(1)); + assertEquals(expectedLinks.get(2), links.get(2)); + assertEquals(expectedLinks.get(3), links.get(3)); + assertEquals(expectedLinks.get(4), links.get(4)); + } + + private List prepareListWithExpectedLinks() { + List expectedLinks = new ArrayList<>(); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/revision_uid?resourceVersion=id%3A176"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/node_type?resourceVersion=id%3A176"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/article"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe"); + + return expectedLinks; + } + + private String prepareContent() { + return "{\"jsonapi\":" + + "{\"version\":\"1.0\",\"meta\":" + + "{\"links\":{\"self\":{\"href\":\"http://jsonapi.org/format/1.0/\"}}}}," + + "\"data\":[" + + "{\"type\":\"node--page\",\"id\":\"e0274815-10ba-4b41-b65f-2e0c918dbe09\"," + + "\"attributes\":{\"drupal_internal__nid\":86},\"" + + "relationships\":{\"node_type\":{\"data\":{\"type\":\"node_type--node_type\",\"id\":\"c142b6c7-af65-4fb5-8fdc-51bdf2b72e92\"}," + + "\"links\":{\"related\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/node_type?resourceVersion=id%3A176\"}," + + "\"self\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/relationships/node_type?resourceVersion=id%3A176\"}}}," + + "\"revision_uid\":{\"data\":{\"type\":\"user--user\",\"id\":\"bfbb1d33-18aa-4726-98e2-3be439022479\"}," + + "\"links\":{\"related\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/revision_uid?resourceVersion=id%3A176\"}}}}}" + + "]," + + "\"links\":{\"node--article\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/article\"}," + + "\"node--page\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page\"}," + + "\"node--recipe\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe\"}," + + "\"self\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion\"}}}"; + } + + @Test(expected = RuntimeException.class) + public void testCollectLinksFromDrupalContent_throwException() { + String url = "http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion"; + String content = prepareWrongContent(); + contentService.collectLinksFromDrupalContent(url, content); + } + + private String prepareWrongContent() { + return "{\"wrongTag\":" + + "{\"version\":\"1.0\",\"meta\":" + + "{\"links\":{\"self\":{\"href\":\"http://jsonapi.org/format/1.0/\"}}}}," + + "\"data\":[]," + + "\"links\":{\"node--article\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/article\"}," + + "\"node--page\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page\"}," + + "\"node--recipe\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe\"}," + + "\"self\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion\"}}}"; + } + } diff --git a/src/test/java/com/lucidworks/fusion/connector/service/DrupalContentCrawlerTest.java b/src/test/java/com/lucidworks/fusion/connector/service/DrupalContentCrawlerTest.java new file mode 100644 index 0000000..dada866 --- /dev/null +++ b/src/test/java/com/lucidworks/fusion/connector/service/DrupalContentCrawlerTest.java @@ -0,0 +1,118 @@ +package com.lucidworks.fusion.connector.service; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.lucidworks.fusion.connector.model.DrupalLoginResponse; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.MockitoAnnotations; +import org.mockito.MockitoSession; +import org.mockito.quality.Strictness; + +import java.util.ArrayList; +import java.util.List; + +import static org.junit.Assert.assertEquals; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.when; + +public class DrupalContentCrawlerTest { + + private static final String URL = "http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion"; + private static final int TOTAL = 5; + + @Mock + private DrupalContentCrawler drupalContentCrawler; + + @Mock + private ContentService contentService; + + @Mock + private DrupalLoginResponse drupalLoginResponse; + + @Mock + private ObjectMapper mapper; + + @Mock + private DrupalOkHttp drupalOkHttp; + + private MockitoSession mockitoSession; + + @Before + public void setUp() { + + mockitoSession = Mockito.mockitoSession() + .initMocks(this) + .strictness(Strictness.LENIENT) + .startMocking(); + MockitoAnnotations.initMocks(this); + + drupalContentCrawler = new DrupalContentCrawler(URL, drupalLoginResponse, contentService, mapper); + } + + @After + public void tearDown() { + mockitoSession.finishMocking(); + } + + @Test + public void testStartCrawling() { + when(drupalOkHttp.getDrupalContent(anyString(), any(DrupalLoginResponse.class))).thenReturn(prepareDrupalContent()); + when(contentService.collectLinksFromDrupalContent(anyString(), anyString())).thenReturn(prepareListWithExpectedLinks()); + drupalContentCrawler.startCrawling(); + + List links = new ArrayList<>(); + links.addAll(drupalContentCrawler.getVisitedUrls().keySet()); + + List expectedLinks = prepareListWithExpectedLinks(); + + assertEquals(TOTAL, links.size()); + assertEquals(expectedLinks.get(0), links.get(0)); + assertEquals(expectedLinks.get(1), links.get(1)); + assertEquals(expectedLinks.get(2), links.get(2)); + assertEquals(expectedLinks.get(3), links.get(3)); + assertEquals(expectedLinks.get(4), links.get(4)); + + } + + private String prepareDrupalContent() { + return "{\"jsonapi\":" + + "{\"version\":\"1.0\",\"meta\":" + + "{\"links\":{\"self\":{\"href\":\"http://jsonapi.org/format/1.0/\"}}}}," + + "\"data\":[" + + "{\"type\":\"node--page\",\"id\":\"e0274815-10ba-4b41-b65f-2e0c918dbe09\"," + + "\"attributes\":{\"drupal_internal__nid\":86},\"" + + "relationships\":{\"node_type\":{\"data\":{\"type\":\"node_type--node_type\",\"id\":\"c142b6c7-af65-4fb5-8fdc-51bdf2b72e92\"}," + + "\"links\":{\"related\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/node_type?resourceVersion=id%3A176\"}," + + "\"self\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/relationships/node_type?resourceVersion=id%3A176\"}}}," + + "\"revision_uid\":{\"data\":{\"type\":\"user--user\",\"id\":\"bfbb1d33-18aa-4726-98e2-3be439022479\"}," + + "\"links\":{\"related\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/revision_uid?resourceVersion=id%3A176\"}}}}}" + + "]," + + "\"links\":{\"node--article\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/article\"}," + + "\"node--page\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page\"}," + + "\"self\":{\"href\":\"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion\"}}}"; + } + + private List prepareListWithExpectedLinks() { + List expectedLinks = new ArrayList<>(); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/revision_uid?resourceVersion=id%3A176"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/article"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page/e0274815-10ba-4b41-b65f-2e0c918dbe09/node_type?resourceVersion=id%3A176"); + expectedLinks.add("http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/page"); + + return expectedLinks; + } + + @Test(expected = RuntimeException.class) + public void testStartCrawling_throwException() { + when(drupalOkHttp.getDrupalContent(any(String.class), any(DrupalLoginResponse.class))).thenReturn(null); + doNothing().when(contentService).collectLinksFromDrupalContent(URL, prepareDrupalContent()); + drupalContentCrawler.startCrawling(); + } + +} diff --git a/src/test/java/com/lucidworks/fusion/connector/util/DataUtilTest.java b/src/test/java/com/lucidworks/fusion/connector/util/DataUtilTest.java new file mode 100644 index 0000000..d9dead7 --- /dev/null +++ b/src/test/java/com/lucidworks/fusion/connector/util/DataUtilTest.java @@ -0,0 +1,378 @@ +package com.lucidworks.fusion.connector.util; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.lucidworks.fusion.connector.exception.ServiceException; +import com.lucidworks.fusion.connector.model.TopLevelJsonApiData; +import com.lucidworks.fusion.connector.model.TopLevelJsonapi; +import org.junit.Before; +import org.junit.Test; + +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class DataUtilTest { + + private ObjectMapper mapper; + + @Before + public void setUp() { + mapper = new ObjectMapper(); + } + + @Test + public void testTopLevelJsonApiFields() { + String content = prepareContent(); + Map topLevelJsonapiMap = new HashMap<>(); + TopLevelJsonapi topLevelJsonapi; + + try { + topLevelJsonapi = mapper.readValue(content, TopLevelJsonapi.class); + + } catch (IOException e) { + try { + topLevelJsonapi = mapper.readValue(content, TopLevelJsonapi.class); + } catch (IOException ex) { + throw new ServiceException("The mapper was unable to read the content!", ex); + } + } + + topLevelJsonapiMap.put("object", topLevelJsonapi); + Map> allObjectsMap = DataUtil.generateObjectMap(topLevelJsonapiMap); + + assertEquals(2, allObjectsMap.size()); + + for (String key : allObjectsMap.keySet()) { + assertTrue(allObjectsMap.get(key).get("html.link").equals(key)); + } + } + + private String prepareContent() { + return "{\n" + + " \"jsonapi\": {\n" + + " \"version\": \"1.0\",\n" + + " \"meta\": {\n" + + " \"links\": {\n" + + " \"self\": {\n" + + " \"href\": \"http://jsonapi.org/format/1.0/\"\n" + + " }\n" + + " }\n" + + " }\n" + + " },\n" + + " \"data\": [\n" + + " {\n" + + " \"type\": \"node--recipe\",\n" + + " \"id\": \"ce732d0e-721a-41a2-9f7e-fa758f75c50d\",\n" + + " \"links\": {\n" + + " \"html\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/recipes/deep-mediterranean-quiche\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d?resourceVersion=id%3A6\"\n" + + " }\n" + + " },\n" + + " \"attributes\": {\n" + + " \"drupal_internal__nid\": 1,\n" + + " \"drupal_internal__vid\": 6,\n" + + " \"langcode\": \"en\",\n" + + " \"revision_timestamp\": \"2020-06-16T07:56:29+00:00\",\n" + + " \"revision_log\": null,\n" + + " \"status\": true,\n" + + " \"title\": \"Deep mediterranean quiche\",\n" + + " \"created\": \"2020-06-16T07:56:29+00:00\",\n" + + " \"changed\": \"2020-06-16T07:56:29+00:00\",\n" + + " \"promote\": true,\n" + + " \"sticky\": false,\n" + + " \"default_langcode\": true,\n" + + " \"revision_translation_affected\": null,\n" + + " \"moderation_state\": \"published\",\n" + + " \"path\": {\n" + + " \"alias\": \"/recipes/deep-mediterranean-quiche\",\n" + + " \"pid\": 331,\n" + + " \"langcode\": \"en\"\n" + + " },\n" + + " \"content_translation_source\": \"und\",\n" + + " \"content_translation_outdated\": false,\n" + + " \"field_cooking_time\": 30,\n" + + " \"field_difficulty\": \"medium\",\n" + + " \"field_ingredients\": [\n" + + " \"For the pastry:\",\n" + + " \"280g plain flour\",\n" + + " \"140g butter\",\n" + + " \"Cold water\",\n" + + " \"For the filling:\",\n" + + " \"1 onion\",\n" + + " \"2 garlic cloves\",\n" + + " \"Half a courgette\",\n" + + " \"450ml soya milk\",\n" + + " \"500g grated parmesan\",\n" + + " \"2 eggs\",\n" + + " \"200g sun dried tomatoes\",\n" + + " \"100g feta\"\n" + + " ],\n" + + " \"field_number_of_servings\": 8,\n" + + " \"field_preparation_time\": 40,\n" + + " \"field_recipe_instruction\": {\n" + + " \"value\": \"
    \\n
  1. Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough.
    2. \\n
  2. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10.
    3. \\n
  3. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette.
    4. \\n
  4. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix.
    5. \\n
  5. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown.
    6. \\n
\\n\",\n" + + " \"format\": \"basic_html\",\n" + + " \"processed\": \"
  1. Preheat the oven to 400°F/200°C. Starting with the pastry; rub the flour and butter together in a bowl until crumbling like breadcrumbs. Add water, a little at a time, until it forms a dough.
    2. \\n
  2. Roll out the pastry on a floured board and gently spread over your tin. Place in the fridge for 20 minutes before blind baking for a further 10.
    3. \\n
  3. Whilst the pastry is cooling, chop and gently cook the onions, garlic and courgette.
    4. \\n
  4. In a large bowl, add the soya milk, half the parmesan, and the eggs. Gently mix.
    5. \\n
  5. Once the pastry is cooked, spread the onions, garlic and sun dried tomatoes over the base and pour the eggs mix over. Sprinkle the remaining parmesan and careful lay the feta over the top. Bake for 30 minutes or until golden brown.
    6. \\n
\"\n" + + " },\n" + + " \"field_summary\": {\n" + + " \"value\": \"An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.\",\n" + + " \"format\": \"basic_html\",\n" + + " \"processed\": \"

An Italian inspired quiche with sun dried tomatoes and courgette. A perfect light meal for a summer's day.

\\n\"\n" + + " }\n" + + " },\n" + + " \"relationships\": {\n" + + " \"node_type\": {\n" + + " \"data\": {\n" + + " \"type\": \"node_type--node_type\",\n" + + " \"id\": \"0bec7ecc-ba15-4907-9011-d6b476329e78\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/node_type?resourceVersion=id%3A6\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/relationships/node_type?resourceVersion=id%3A6\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"revision_uid\": {\n" + + " \"data\": {\n" + + " \"type\": \"user--user\",\n" + + " \"id\": \"a3ded394-12db-4e25-8d70-37fd96ccaf89\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/revision_uid?resourceVersion=id%3A6\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/relationships/revision_uid?resourceVersion=id%3A6\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"uid\": {\n" + + " \"data\": {\n" + + " \"type\": \"user--user\",\n" + + " \"id\": \"a3ded394-12db-4e25-8d70-37fd96ccaf89\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/uid?resourceVersion=id%3A6\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/relationships/uid?resourceVersion=id%3A6\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"field_media_image\": {\n" + + " \"data\": {\n" + + " \"type\": \"media--image\",\n" + + " \"id\": \"7e43e064-dbd4-4245-8541-22d63c8cdaa5\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/field_media_image?resourceVersion=id%3A6\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/relationships/field_media_image?resourceVersion=id%3A6\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"field_recipe_category\": {\n" + + " \"data\": [\n" + + " {\n" + + " \"type\": \"taxonomy_term--recipe_category\",\n" + + " \"id\": \"f9cd39e6-5e8a-4ec7-83b0-edfa0ad74c14\"\n" + + " }\n" + + " ],\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/field_recipe_category?resourceVersion=id%3A6\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/relationships/field_recipe_category?resourceVersion=id%3A6\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"field_tags\": {\n" + + " \"data\": [\n" + + " {\n" + + " \"type\": \"taxonomy_term--tags\",\n" + + " \"id\": \"1ee898a9-da19-46c7-b7cf-02e20c666c64\"\n" + + " },\n" + + " {\n" + + " \"type\": \"taxonomy_term--tags\",\n" + + " \"id\": \"ba07f0e3-ef59-4feb-bf8a-fb9914782992\"\n" + + " }\n" + + " ],\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/field_tags?resourceVersion=id%3A6\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/ce732d0e-721a-41a2-9f7e-fa758f75c50d/relationships/field_tags?resourceVersion=id%3A6\"\n" + + " }\n" + + " }\n" + + " }\n" + + " }\n" + + " }," + + "\n" + + " {\n" + + " \"type\": \"node--recipe\",\n" + + " \"id\": \"6671c26c-e325-42a5-b501-d2cdddb612a8\",\n" + + " \"links\": {\n" + + " \"html\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/node/956\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8?resourceVersion=id%3A1441\"\n" + + " }\n" + + " },\n" + + " \"attributes\": {\n" + + " \"drupal_internal__nid\": 956,\n" + + " \"drupal_internal__vid\": 1441,\n" + + " \"langcode\": \"en\",\n" + + " \"revision_timestamp\": \"2020-06-26T13:03:33+00:00\",\n" + + " \"revision_log\": null,\n" + + " \"status\": true,\n" + + " \"title\": \"Vegan chocolate and nut brownies\",\n" + + " \"created\": \"2020-06-26T13:02:20+00:00\",\n" + + " \"changed\": \"2020-06-26T13:03:33+00:00\",\n" + + " \"promote\": true,\n" + + " \"sticky\": false,\n" + + " \"default_langcode\": true,\n" + + " \"revision_translation_affected\": true,\n" + + " \"moderation_state\": \"published\",\n" + + " \"path\": {\n" + + " \"alias\": null,\n" + + " \"pid\": null,\n" + + " \"langcode\": \"en\"\n" + + " },\n" + + " \"content_translation_source\": \"und\",\n" + + " \"content_translation_outdated\": false,\n" + + " \"field_cooking_time\": 23,\n" + + " \"field_difficulty\": \"hard\",\n" + + " \"field_ingredients\": [\n" + + " \"6 tbsp sunflower oil\",\n" + + " \"80g vegan dark chocolate\",\n" + + " \"170g plain flour\",\n" + + " \"1 tsp baking powder\"\n" + + " ],\n" + + " \"field_number_of_servings\": 4,\n" + + " \"field_preparation_time\": 45,\n" + + " \"field_recipe_instruction\": {\n" + + " \"value\": \"
    \\r\\n\\t
  1. Use a little of the sunflower oil to grease an 8 inch square baking tin (or similar size) and line the tin with greaseproof paper.
    2. \\r\\n\\t
  2. Preheat the oven to 350°F/180°C.
    3. \\r\\n\\t
  3. Break approximately 1/3rd of the chocolate bar off and chop into small pieces. Roughly chop 2/3rds of the pecan nuts and mix together with the chopped chocolate. Set aside.
    4. \\r\\n\\t
  4. For finishing the brownies, chop or crush the remaining pecan nuts and walnuts, mix together and set aside.
    5. \\r\\n\\t
  5. Melt the remaining chocolate by bringing a couple inches of water to the boil in a small saucepan that is suitably sized for holding a heatproof bowl in the pan opening. Do not allow the bottom of the heatproof bowl to touch the water. Place the chocolate into the bowl to melt, stirring occasionally to ensure the chocolate has fully melted. Once melted, set aside and allow to cool slightly.
    6. \\r\\n\\t
  6. Whilst the chocolate is melting, begin to sieve the plain flour, coconut flour, and cocoa powder into a large mixing bowl and mix. Once mixed, stir in the baking powder and sugar.
    7. \\r\\n\\t
  7. Once the chocolate has cooled a little, begin to slowly stir the vanilla essence, sunflower oil, soya milk, and melted chocolate into the flour and cocoa mix.
    8. \\r\\n\\t
  8. Now stir in the previously chopped chocolate and pecan nuts, ensuring they are stirred evenly into the mixture.
    9. \\r\\n\\t
  9. Pour the mixture into the baking tin and spread evenly with a spatula.
    10. \\r\\n\\t
  10. Sprinkle the chopped pecan nuts and walnuts across the top and bake in the centre of the oven for 18 to 23 minutes.
    11. \\r\\n\\t
  11. Remove from the oven and allow to cool for 45 minutes. Carefully use the edges of the greaseproof paper to lift the brownie out of the tin and place onto a chopping board. With a sharp knife, gently cut into evenly sized pieces.
    12. \\r\\n\\t
  12. Serve on their own or with some vegan cream or ice cream.
    13. \\r\\n
\\r\\n\",\n" + + " \"format\": \"basic_html\",\n" + + " \"processed\": \"
  1. Use a little of the sunflower oil to grease an 8 inch square baking tin (or similar size) and line the tin with greaseproof paper.
    2. \\n
  2. Preheat the oven to 350°F/180°C.
    3. \\n
  3. Break approximately 1/3rd of the chocolate bar off and chop into small pieces. Roughly chop 2/3rds of the pecan nuts and mix together with the chopped chocolate. Set aside.
    4. \\n
  4. For finishing the brownies, chop or crush the remaining pecan nuts and walnuts, mix together and set aside.
    5. \\n
  5. Melt the remaining chocolate by bringing a couple inches of water to the boil in a small saucepan that is suitably sized for holding a heatproof bowl in the pan opening. Do not allow the bottom of the heatproof bowl to touch the water. Place the chocolate into the bowl to melt, stirring occasionally to ensure the chocolate has fully melted. Once melted, set aside and allow to cool slightly.
    6. \\n
  6. Whilst the chocolate is melting, begin to sieve the plain flour, coconut flour, and cocoa powder into a large mixing bowl and mix. Once mixed, stir in the baking powder and sugar.
    7. \\n
  7. Once the chocolate has cooled a little, begin to slowly stir the vanilla essence, sunflower oil, soya milk, and melted chocolate into the flour and cocoa mix.
    8. \\n
  8. Now stir in the previously chopped chocolate and pecan nuts, ensuring they are stirred evenly into the mixture.
    9. \\n
  9. Pour the mixture into the baking tin and spread evenly with a spatula.
    10. \\n
  10. Sprinkle the chopped pecan nuts and walnuts across the top and bake in the centre of the oven for 18 to 23 minutes.
    11. \\n
  11. Remove from the oven and allow to cool for 45 minutes. Carefully use the edges of the greaseproof paper to lift the brownie out of the tin and place onto a chopping board. With a sharp knife, gently cut into evenly sized pieces.
    12. \\n
  12. Serve on their own or with some vegan cream or ice cream.
    13. \\n
\"\n" + + " },\n" + + " \"field_summary\": {\n" + + " \"value\": \"

Scrumptious vegan chocolate brownies that are rich, fudgy, and nutty. These delights have a surprise hint of coconut making them the perfect indulgence. Serve warm with a little vanilla dairy-free ice cream!

\\r\\n\",\n" + + " \"format\": \"basic_html\",\n" + + " \"processed\": \"

Scrumptious vegan chocolate brownies that are rich, fudgy, and nutty. These delights have a surprise hint of coconut making them the perfect indulgence. Serve warm with a little vanilla dairy-free ice cream!

\\n\"\n" + + " }\n" + + " },\n" + + " \"relationships\": {\n" + + " \"node_type\": {\n" + + " \"data\": {\n" + + " \"type\": \"node_type--node_type\",\n" + + " \"id\": \"0bec7ecc-ba15-4907-9011-d6b476329e78\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/node_type?resourceVersion=id%3A1441\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/relationships/node_type?resourceVersion=id%3A1441\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"revision_uid\": {\n" + + " \"data\": {\n" + + " \"type\": \"user--user\",\n" + + " \"id\": \"37444321-dc8f-434f-871e-be7a9a9ecd44\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/revision_uid?resourceVersion=id%3A1441\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/relationships/revision_uid?resourceVersion=id%3A1441\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"uid\": {\n" + + " \"data\": {\n" + + " \"type\": \"user--user\",\n" + + " \"id\": \"37444321-dc8f-434f-871e-be7a9a9ecd44\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/uid?resourceVersion=id%3A1441\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/relationships/uid?resourceVersion=id%3A1441\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"field_media_image\": {\n" + + " \"data\": {\n" + + " \"type\": \"media--image\",\n" + + " \"id\": \"ef0d2ae6-4ae3-41c5-90aa-ace022c53b46\"\n" + + " },\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/field_media_image?resourceVersion=id%3A1441\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/relationships/field_media_image?resourceVersion=id%3A1441\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"field_recipe_category\": {\n" + + " \"data\": [\n" + + " {\n" + + " \"type\": \"taxonomy_term--recipe_category\",\n" + + " \"id\": \"8b2ffca7-e7b5-4151-94f5-5595044326ab\"\n" + + " }\n" + + " ],\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/field_recipe_category?resourceVersion=id%3A1441\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/relationships/field_recipe_category?resourceVersion=id%3A1441\"\n" + + " }\n" + + " }\n" + + " },\n" + + " \"field_tags\": {\n" + + " \"data\": [\n" + + " {\n" + + " \"type\": \"taxonomy_term--tags\",\n" + + " \"id\": \"bed81a5a-ec6c-41a3-9ec0-a7cc2e3d19e9\"\n" + + " }\n" + + " ],\n" + + " \"links\": {\n" + + " \"related\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/field_tags?resourceVersion=id%3A1441\"\n" + + " },\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe/6671c26c-e325-42a5-b501-d2cdddb612a8/relationships/field_tags?resourceVersion=id%3A1441\"\n" + + " }\n" + + " }\n" + + " }\n" + + " }\n" + + " }\n" + + " ],\n" + + " \"links\": {\n" + + " \"self\": {\n" + + " \"href\": \"http://s5ee7c4bb7c413wcrxueduzw.devcloud.acquia-sites.com/en/fusion/node/recipe\"\n" + + " }\n" + + " }" + + " }"; + } +}