Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LF-4625a Update controller for sensor addition #3655

Open
wants to merge 5 commits into
base: integration
Choose a base branch
from
+226 −211
Open

LF-4625a Update controller for sensor addition #3655

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ae6ab03
LF-4625 Refactor Ensemble Scientific portion of addSensors controller…
kathyavini Jan 3, 2025
61cb14c
LF-4625 Update registerOrganizationWebhook and bulkSensorClaim for re…
kathyavini Jan 3, 2025
8503f81
LF-4625 Remove CSV parsing and take data directly from req.body
kathyavini Jan 7, 2025
b0f0056
LF-4625 Add functionality to associate farm with Ensemble organizatio…
kathyavini Jan 16, 2025
b976948
LF-4625 Rename controller and endpoint to use 'linking' terminology f…
kathyavini Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
332 changes: 146 additions & 186 deletions packages/api/src/controllers/sensorController.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,44 +23,23 @@ import FarmExternalIntegrationsModel from '../models/farmExternalIntegrationsMod
import LocationModel from '../models/locationModel.js';
import PointModel from '../models/pointModel.js';
import FigureModel from '../models/figureModel.js';
import UserModel from '../models/userModel.js';
import PartnerReadingTypeModel from '../models/PartnerReadingTypeModel.js';

import { transaction, Model } from 'objection';

import {
createOrganization,
registerOrganizationWebhook,
bulkSensorClaim,
ENSEMBLE_BRAND,
getEnsembleOrganizations,
extractEsids,
registerFarmAndClaimSensors,
unclaimSensor,
ENSEMBLE_UNITS_MAPPING,
} from '../util/ensemble.js';
import { databaseUnit } from '../util/unit.js';
import { sensorErrors, parseSensorCsv } from '../../../shared/validation/sensorCSV.js';
import { sensorErrors } from '../../../shared/validation/sensorCSV.js';
import syncAsyncResponse from '../util/syncAsyncResponse.js';
import knex from '../util/knex.js';

const getSensorTranslations = async (language) => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason I am hesitant to just partially remove the CSV option haha. It mostly works but just doesn't meet ideal design and needs to be adapted for new fields maybe. I can't deny I am partial to leaving it in place for custom sensors 🥲

And maybe editing sensor controller right now won't be necessary if you agree that it is rest-ful to work on the /farm_external_integration resource instead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll defer to @antsgar on the CSV upload; there will be no way to access it anymore from app is why I think it's okay to remove now, but you're right it's a partial removal as I only adjusted in this particular controller. That was just my understanding of the ticket requirements, but really there is a lot more code that can be deleted (from shared and webapp) if we are wiping CSV upload entirely. That deletion might be best in its own PR so even if we rebuild CSV addition from the ground up when we re-introduce it (which is what I believe we're going to do), we'll have just one place to look for all of the original code.

// Remove country identifier from language preference
const parsedLanguage = language.includes('-') ? language.split('-')[0] : language;
let translations;
try {
translations = await import(`../../../shared/locales/${parsedLanguage}/sensorCSV.json`, {
assert: { type: 'json' },
});
// Default to english in case where user language not supported
if (!translations) {
throw 'Translations not found';
}
} catch (error) {
console.log(error);
translations = await import(`../../../shared/locales/en/sensorCSV.json`, {
assert: { type: 'json' },
});
}
return translations.default;
};

const sensorController = {
async getSensorReadingTypes(req, res) {
const { location_id } = req.params;
Expand Down Expand Up @@ -106,6 +85,43 @@ const sensorController = {
res.status(404).send('Partner not found');
}
},
async associateEnsembleOrganization(req, res) {
const { farm_id } = req.headers;
const { organization_uuid } = req.body;

if (!organization_uuid || !organization_uuid.length) {
return res.status(400).send('Organization uuid required');
}

try {
const { access_token } = await IntegratingPartnersModel.getAccessAndRefreshTokens(
ENSEMBLE_BRAND,
);

const allRegisteredOrganizations = await getEnsembleOrganizations(access_token);

const organization = allRegisteredOrganizations.find(
({ uuid }) => uuid === organization_uuid,
Copy link
Collaborator Author

@kathyavini kathyavini Jan 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where I would very much like to see a secondary check, maybe on email or farm name? But we would have to coordinate with Ensemble to make sure their organization registration included it.

);

if (!organization) {
return res.status(404).send('Organization not found');
}

await FarmExternalIntegrationsModel.upsertOrganizationIntegration({
farm_id,
partner_id: 1,
organization_uuid,
});

return res.status(200).send();
} catch (error) {
console.log(error);
return res.status(400).json({
error,
});
}
},
async addSensors(req, res) {
let timeLimit = 5000;
const testTimerOverride = Number(req.query?.sensorUploadTimer);
Expand All @@ -119,23 +135,105 @@ const sensorController = {
const { user_id } = req.auth;
try {
const { access_token } = await IntegratingPartnersModel.getAccessAndRefreshTokens(
'Ensemble Scientific',
ENSEMBLE_BRAND,
);

//TODO: LF-4443 - Sensor should not use User language (unrestricted string), accept as body param or farm level detail
const [{ language_preference }] = await baseController.getIndividual(UserModel, user_id);
const data = req.body;

const translations = await getSensorTranslations(language_preference);
const { data, errors } = parseSensorCsv(
req.file.buffer.toString(),
language_preference,
translations,
);
// Extract Ensemble Scientific sensor IDs (esids)
const esids = extractEsids(data);

let success = [];
let already_owned = [];
let does_not_exist = [];
let occupied = [];

if (esids.length > 0) {
({ success, already_owned, does_not_exist, occupied } = await registerFarmAndClaimSensors(
farm_id,
access_token,
esids,
));
}

const registeredSensors = [];
const errorSensors = [];

// Iterate over each sensor in the data array
data.forEach((sensor, index) => {
if (sensor.brand !== ENSEMBLE_BRAND) {
// All non-ESCI sensors should be considered successfully registered
registeredSensors.push(sensor);
} else if (
success.includes(sensor.external_id) ||
already_owned.includes(sensor.external_id)
) {
registeredSensors.push(sensor);
} else if (does_not_exist.includes(sensor.external_id)) {
errorSensors.push({
row: index + 2,
column: 'External_ID',
translation_key: sensorErrors.SENSOR_DOES_NOT_EXIST,
variables: { sensorId: sensor.external_id },
});
} else if (occupied.includes(sensor.external_id)) {
errorSensors.push({
row: index + 2,
column: 'External_ID',
translation_key: sensorErrors.SENSOR_ALREADY_OCCUPIED,
variables: { sensorId: sensor.external_id },
});
} else {
// We know that it is an ESID but it was not returned in the expected format from the API
errorSensors.push({
row: index + 2,
column: 'External_ID',
translation_key: sensorErrors.INTERNAL_ERROR,
variables: { sensorId: sensor.external_id },
});
}
});

// Save sensors in database
const sensorLocations = [];
for (const sensor of registeredSensors) {
try {
const value = await SensorModel.createSensor(
sensor,
farm_id,
user_id,
esids.includes(sensor.external_id) ? 1 : 0,
);
sensorLocations.push({ status: 'fulfilled', value });
} catch (error) {
sensorLocations.push({ status: 'rejected', reason: error });
}
}

const successSensors = sensorLocations.reduce((prev, curr, idx) => {
if (curr.status === 'fulfilled') {
prev.push(curr.value);
} else {
// These are sensors that were not saved to the database as locations
errorSensors.push({
row: data.findIndex((elem) => elem === registeredSensors[idx]) + 2,
translation_key: sensorErrors.INTERNAL_ERROR,
variables: {
sensorId: registeredSensors[idx].external_id || registeredSensors[idx].name,
},
});
}
return prev;
}, []);

if (errors.length > 0) {
return await sendResponse(
if (successSensors.length < data.length) {
return sendResponse(
() => {
return res.status(400).send({ error_type: 'validation_failure', errors });
return res.status(400).send({
error_type: 'unable_to_claim_all_sensors',
success: successSensors, // We need the full sensor objects to update the redux store
errorSensors,
});
},
async () => {
return await sendSensorNotification(
Expand All @@ -144,168 +242,30 @@ const sensorController = {
SensorNotificationTypes.SENSOR_BULK_UPLOAD_FAIL,
{
error_download: {
errors,
errors: errorSensors,
file_name: 'sensor-upload-outcomes.txt',
error_type: 'validation',
success: successSensors.map((s) => s.sensor?.external_id || s.name), // Notification download needs an array of only ESIDs
error_type: 'claim',
},
},
);
},
);
} else if (!data.length > 0) {
return await sendResponse(
} else {
return sendResponse(
() => {
return res.status(400).send({ error_type: 'empty_file' });
return res
.status(200)
.send({ message: 'Successfully uploaded!', sensors: successSensors });
},
async () => {
return await sendSensorNotification(
user_id,
farm_id,
SensorNotificationTypes.SENSOR_BULK_UPLOAD_FAIL,
{
error_download: {
errors: [],
file_name: 'sensor-upload-outcomes.txt',
error_type: 'generic',
},
},
SensorNotificationTypes.SENSOR_BULK_UPLOAD_SUCCESS,
);
},
);
} else {
const esids = data.reduce((previous, current) => {
if (current.brand === 'Ensemble Scientific' && current.external_id) {
previous.push(current.external_id);
}
return previous;
}, []);
let success = [];
let already_owned = [];
let does_not_exist = [];
let occupied = [];
if (esids.length > 0) {
const organization = await createOrganization(farm_id, access_token);

// register webhook for sensor readings
await registerOrganizationWebhook(farm_id, organization.organization_uuid, access_token);

// Register sensors with Ensemble
({ success, already_owned, does_not_exist, occupied } = await bulkSensorClaim(
access_token,
organization.organization_uuid,
esids,
));
}
// register organization

// Filter sensors by those successfully registered and those with errors
const { registeredSensors, errorSensors } = data.reduce(
(prev, curr, idx) => {
if (success?.includes(curr.external_id) || already_owned?.includes(curr.external_id)) {
prev.registeredSensors.push(curr);
} else if (curr.brand !== 'Ensemble Scientific') {
prev.registeredSensors.push(curr);
} else if (does_not_exist?.includes(curr.external_id)) {
prev.errorSensors.push({
row: idx + 2,
column: 'External_ID',
translation_key: sensorErrors.SENSOR_DOES_NOT_EXIST,
variables: { sensorId: curr.external_id },
});
} else if (occupied?.includes(curr.external_id)) {
prev.errorSensors.push({
row: idx + 2,
column: 'External_ID',
translation_key: sensorErrors.SENSOR_ALREADY_OCCUPIED,
variables: { sensorId: curr.external_id },
});
} else {
// we know that it is an ESID but for some reason it was not returned in the expected format from the API
prev.errorSensors.push({
row: idx + 2,
column: 'External_ID',
translation_key: sensorErrors.INTERNAL_ERROR,
variables: { sensorId: curr.external_id },
});
}
return prev;
},
{ registeredSensors: [], errorSensors: [] },
);

// Save sensors in database
const sensorLocations = [];
for (const sensor of registeredSensors) {
try {
const value = await SensorModel.createSensor(
sensor,
farm_id,
user_id,
esids.includes(sensor.external_id) ? 1 : 0,
);
sensorLocations.push({ status: 'fulfilled', value });
} catch (error) {
sensorLocations.push({ status: 'rejected', reason: error });
}
}

const successSensors = sensorLocations.reduce((prev, curr, idx) => {
if (curr.status === 'fulfilled') {
prev.push(curr.value);
} else {
// These are sensors that were not saved to the database as locations
errorSensors.push({
row: data.findIndex((elem) => elem === registeredSensors[idx]) + 2,
translation_key: sensorErrors.INTERNAL_ERROR,
variables: {
sensorId: registeredSensors[idx].external_id || registeredSensors[idx].name,
},
});
}
return prev;
}, []);

if (successSensors.length < data.length) {
return sendResponse(
() => {
return res.status(400).send({
error_type: 'unable_to_claim_all_sensors',
success: successSensors, // We need the full sensor objects to update the redux store
errorSensors,
});
},
async () => {
return await sendSensorNotification(
user_id,
farm_id,
SensorNotificationTypes.SENSOR_BULK_UPLOAD_FAIL,
{
error_download: {
errors: errorSensors,
file_name: 'sensor-upload-outcomes.txt',
success: successSensors.map((s) => s.sensor?.external_id || s.name), // Notification download needs an array of only ESIDs
error_type: 'claim',
},
},
);
},
);
} else {
return sendResponse(
() => {
return res
.status(200)
.send({ message: 'Successfully uploaded!', sensors: successSensors });
},
async () => {
return await sendSensorNotification(
user_id,
farm_id,
SensorNotificationTypes.SENSOR_BULK_UPLOAD_SUCCESS,
);
},
);
}
}
} catch (e) {
console.log(e);
Expand Down Expand Up @@ -630,7 +590,7 @@ const sensorController = {

const user_id = req.auth.user_id;
const { access_token } = await IntegratingPartnersModel.getAccessAndRefreshTokens(
'Ensemble Scientific',
ENSEMBLE_BRAND,
);
let unclaimResponse;
if (partner_name != 'No Integrating Partner' && external_id != '') {
Expand Down
Loading
Loading