Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#2350: Generate survey files async #2362

Merged
merged 46 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
100784f
create SurveyFilesManager actor to handle generation of files
anaPerezGhiglia Aug 1, 2024
1539c01
Generate interactions file into a new dir
anaPerezGhiglia Aug 1, 2024
aa163c8
add basic decision for generating file or not
anaPerezGhiglia Aug 1, 2024
d39129e
Add incentives file to SurveyFilesManager
anaPerezGhiglia Aug 2, 2024
c4c22ef
add disposition_history files to SurveyFilesManager
anaPerezGhiglia Aug 2, 2024
3a3d084
Convert csv trigger request into POST requests with no csv format
anaPerezGhiglia Aug 3, 2024
d1f861b
add respondents_result file to SurveyFilesManager
anaPerezGhiglia Aug 3, 2024
400135d
Cleanup
anaPerezGhiglia Aug 5, 2024
45b1181
Fix router path to point to new get_respondents_result path
anaPerezGhiglia Aug 5, 2024
59f4893
Remove respondents_where from Survey model
anaPerezGhiglia Aug 5, 2024
bba89be
Rename Ask.Runtime.SurveyFilesManager to Ask.SurveyResults
anaPerezGhiglia Aug 5, 2024
65e6f12
Move all_questionnaires_fields to SurveyResults
anaPerezGhiglia Aug 5, 2024
2017114
remove unused import
anaPerezGhiglia Aug 5, 2024
1264365
Scaffold Survey Results tests
matiasgarciaisaia Aug 13, 2024
346e866
More tests
matiasgarciaisaia Aug 14, 2024
2244b16
Keep moving tests
matiasgarciaisaia Aug 14, 2024
2cc0605
Make test suite "pass" by skipping broken tests
matiasgarciaisaia Aug 15, 2024
744e17a
File downloads UI - first take
matiasgarciaisaia Sep 11, 2024
1f50e33
Stub downloading previously generated result files
matiasgarciaisaia Sep 24, 2024
af1a87e
Typo
matiasgarciaisaia Sep 24, 2024
8d0f518
Generate async files in a stable path
matiasgarciaisaia Sep 25, 2024
17496f6
Trigger files generation
matiasgarciaisaia Sep 25, 2024
014cb13
Statically serve the generated files
matiasgarciaisaia Sep 26, 2024
c76fada
Add file status endpoint
matiasgarciaisaia Oct 1, 2024
0d545fd
Make API return a map of files instead of an array
matiasgarciaisaia Oct 2, 2024
8ef0ffd
Fetch files status upon Downloads dialog load
matiasgarciaisaia Oct 2, 2024
1b17181
Fetch files status upon download dialog
matiasgarciaisaia Oct 14, 2024
b14a27a
Enable/disable file buttons depending on state
matiasgarciaisaia Oct 29, 2024
d3e1923
Regularly fetch respondent files status
matiasgarciaisaia Oct 31, 2024
1b7d329
Fix SurveyResults tests
matiasgarciaisaia Nov 5, 2024
2dd4b58
Fixing FlowJS typing
matiasgarciaisaia Nov 5, 2024
bacac9c
Ignore Flow-typing react-timeago
matiasgarciaisaia Nov 6, 2024
2526d5e
Remove unused variables
matiasgarciaisaia Nov 6, 2024
b8fd21c
Drop old FIXMEs
matiasgarciaisaia Nov 7, 2024
e02a40b
Log file download vs generation in ActivityLog
matiasgarciaisaia Nov 7, 2024
d95502b
Style Download CSV files modal
matiasgarciaisaia Nov 12, 2024
d3e967d
Access control generated CSV files
matiasgarciaisaia Nov 13, 2024
b04f065
Move API routes concerns back to api.js
matiasgarciaisaia Nov 13, 2024
bb8a8ac
Fix: filter was not updated in file status queries
matiasgarciaisaia Nov 13, 2024
d5ceb9a
Unskip survey links tests
matiasgarciaisaia Nov 13, 2024
6c1d42e
Respond a 404 for files not yet generated
matiasgarciaisaia Nov 14, 2024
c9ba84b
Fix: short links point to the right file downloads
matiasgarciaisaia Nov 14, 2024
63a3d36
Honour results filter when generating file
matiasgarciaisaia Dec 4, 2024
b3414c3
Nitpick: renames and comments from the PR
matiasgarciaisaia Dec 5, 2024
6ad6f92
Fix: avoid fetching file status after navigating
matiasgarciaisaia Dec 9, 2024
9b2b9c5
Mouse pointer on file generation button
matiasgarciaisaia Dec 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assets/css/_buttons.scss
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ i {
@extend .grey-text;
font-size: 17px;
display: inline-block;
margin-left: 10px;
margin-left: 10px; // TODO: the spacing should always be placed in the container, not in the content
cursor: pointer;
span {
vertical-align: middle;
Expand Down
107 changes: 61 additions & 46 deletions assets/css/_card-modal.scss
Original file line number Diff line number Diff line change
Expand Up @@ -74,49 +74,11 @@
li.collection-item {
border: none;
}
li.collection-item.download {
min-height: 6rem;
}
a.download {
@extend .grey-text;
position: relative;
> div {
padding-left: 70px;
max-width: 85%;
&:first-child {
background-color: #999999;
text-align: center;
border-radius: 50%;
height: 45px;
width: 45px;
padding-left: 0;
max-width: initial;
position: absolute;
top: 4px;
left: 5px;
i.material-icons {
@extend .white-text;
font-size: 1.5rem;
line-height: 45px;
}
}
p {
font-size: 1.1rem;
margin: 0.2rem 0;
}
}
&:after {
content: "";
display: table;
clear: both;
zoom: 1;
}
.button {
background-color: color("grey", "darken-1");
}
.button[disabled] {
border: 1px rgba(0, 0, 0, 0.1) solid;
background-color: transparent;
li.collection-item .file-section {
padding-left: 70px;
p {
font-size: 1.1rem;
margin: 0.2rem 0;
}
.title {
@extend .black-text;
Expand All @@ -132,19 +94,48 @@
width: fit-content;
}
}
a.download {
display: block;
text-align: center;
height: 45px;
width: 45px;
padding-left: 0;
max-width: initial;
i.material-icons {
@extend .black-text;
font-size: 1.5rem;
}
position: absolute;
top: 4px;
left: 5px;
&:after {
content: "";
display: table;
clear: both;
zoom: 1;
}
}
.access-link {
padding: 0.5rem 0;
max-width: 85%;
display: flex;
align-items: center;
.switch {
display: inline-block;
vertical-align: middle;
display: flex;
align-items: center;
gap: 0.5rem;
label {
display: flex;
align-items: center;
}
.lever {
margin-left: 0;
}
.label {
@extend .grey-text;
font-size: 1rem;
margin-right: 0.5rem;
display: flex;
white-space: nowrap;
}
}
.link {
Expand All @@ -160,6 +151,7 @@
vertical-align: middle;
width: 98%;
overflow: hidden;
text-overflow: ellipsis;
display: inline-block;
}
.buttons {
Expand All @@ -179,6 +171,29 @@
}
}
}
.file-download {
display: flex;
align-items: center;
gap: 0.5rem;

.btn-icon-grey {
margin-left: initial;
}

.file-generation {
display: flex;
align-items: center;
gap: 0.5rem;
}

a {
display: inline-flex;
}

.material-icons {
display: inline-flex;
}
}
div.link {
@extend .grey-text;
> div {
Expand Down
57 changes: 57 additions & 0 deletions assets/js/actions/survey.js
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ export const REFRESH_LINK = "SURVEY_REFRESH_LINK"
export const DELETE_LINK = "SURVEY_DELETE_LINK"
export const RECEIVE_SURVEY_STATS = "RECEIVE_SURVEY_STATS"
export const RECEIVE_SURVEY_RETRIES_HISTOGRAMS = "RECEIVE_SURVEY_RETRIES_HISTOGRAMS"
export const GENERATING_FILE = "GENERATING_FILE"
export const FETCHING_FILES_STATUS = "FETCHING_FILES_STATUS"
export const RECEIVE_FILES_STATUS = "RECEIVE_FILES_STATUS"

export const createSurvey =
(projectId: number, folderId?: number) => (dispatch: Function, getState: () => Store) =>
Expand Down Expand Up @@ -366,6 +369,11 @@ export const deleteLink = (link: Link) => ({
link,
})

export const generatingFile = (file: string) => ({
type: GENERATING_FILE,
file
})

export const createResultsLink = (projectId: number, surveyId: number) => (dispatch: Function) => {
api.createResultsLink(projectId, surveyId).then((response) => {
return dispatch(receiveLink(response))
Expand Down Expand Up @@ -448,3 +456,52 @@ export const deleteDispositionHistoryLink =
return dispatch(deleteLink(link))
})
}

export const fetchingRespondentFilesStatus = (surveyId: number) => ({
type: FETCHING_FILES_STATUS,
surveyId,
})

export const receiveRespondentsFilesStatus = (surveyId: number, surveyState: string, files: SurveyFiles) => ({
type: RECEIVE_FILES_STATUS,
surveyId,
surveyState,
files,
})

export const fetchRespondentsFilesStatus = (projectId: number, surveyId: number, filter?: string) => (dispatch: Function) => {
dispatch(fetchingRespondentFilesStatus(surveyId))
api.fetchRespondentsFilesStatus(projectId, surveyId, filter).then((response) => {
dispatch(receiveRespondentsFilesStatus(surveyId, response.survey_state, response.files))
})
}

export const generateResultsFile =
(projectId: number, surveyId: number, filter?: string) => (dispatch: Function) => {
api.generateResults(projectId, surveyId, filter).then((response) => {
return dispatch(generatingFile("respondent-results"))
})
}

export const generateIncentivesFile =
(projectId: number, surveyId: number) => (dispatch: Function) => {
// TODO: better handle when incentives download are disabled (due to sample with IDs)
matiasgarciaisaia marked this conversation as resolved.
Show resolved Hide resolved
api.generateIncentives(projectId, surveyId).then((response) => {
return dispatch(generatingFile("incentives"))
})
}

export const generateInteractionsFile =
(projectId: number, surveyId: number) => (dispatch: Function) => {
api.generateInteractions(projectId, surveyId).then((response) => {
return dispatch(generatingFile("interactions"))
})
}

export const generateDispositionHistoryFile =
(projectId: number, surveyId: number) => (dispatch: Function) => {
api.generateDispositionHistory(projectId, surveyId).then((response) => {
return dispatch(generatingFile("disposition-history"))
})
}

26 changes: 19 additions & 7 deletions assets/js/api.js
Original file line number Diff line number Diff line change
Expand Up @@ -691,17 +691,29 @@ export const refreshDispositionHistoryLink = (projectId, surveyId) => {
)
}

export const triggerRespondentsResultFile = (projectId, surveyId, q) =>
apiPostJSON(`projects/${projectId}/surveys/${surveyId}/respondents/results?${
(q && `&q=${encodeURIComponent(q)}`) || ""
}`, null, null)
export const triggerRespondentsDispositionHistoryCSV = (projectId, surveyId) =>
export const resultsFileUrl = (projectId, surveyId, q) =>
`/api/v1/projects/${projectId}/surveys/${surveyId}/respondents/results_csv${
(q && `?q=${encodeURIComponent(q)}`) || ""
}`
export const dispositionHistoryFileUrl = (projectId, surveyId) =>
`/api/v1/projects/${projectId}/surveys/${surveyId}/respondents/disposition_history`
export const incentivesFileUrl = (projectId, surveyId) =>
`/api/v1/projects/${projectId}/surveys/${surveyId}/respondents/incentives`
export const interactionsFileUrl = (projectId, surveyId) =>
`/api/v1/projects/${projectId}/surveys/${surveyId}/respondents/interactions`

export const generateResults = (projectId, surveyId, filter) =>
apiPostJSON(`projects/${projectId}/surveys/${surveyId}/respondents/results`, null, { q: filter })
export const generateDispositionHistory = (projectId, surveyId) =>
apiPostJSON(`projects/${projectId}/surveys/${surveyId}/respondents/disposition_history`, null, null)
export const triggerRespondentsIncentivesCSV = (projectId, surveyId) =>
export const generateIncentives = (projectId, surveyId) =>
apiPostJSON(`projects/${projectId}/surveys/${surveyId}/respondents/incentives`, null, null)
export const triggerRespondentsInteractionsCSV = (projectId, surveyId) =>
export const generateInteractions = (projectId, surveyId) =>
apiPostJSON(`projects/${projectId}/surveys/${surveyId}/respondents/interactions`, null, null)

export const fetchRespondentsFilesStatus = (projectId, surveyId, filter) =>
apiFetch(`projects/${projectId}/surveys/${surveyId}/respondents/files?q=${filter}`).then((response) => response.json())

export const startSimulation = (projectId, questionnaireId, mode) => {
return apiPutOrPostJSONWithCallback(
`projects/${projectId}/questionnaires/${questionnaireId}/simulation`,
Expand Down
5 changes: 5 additions & 0 deletions assets/js/components/activity/ActivityDescription.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,11 @@ class ActivityDescription extends Component {
surveyName: surveyName,
reportType: reportType,
})
case "generate_file":
return t("Generated <i>{{surveyName}}</i> {{reportType}} file", {
surveyName: surveyName,
reportType: reportType,
})
case "enable_public_link":
return t("Enabled <i>{{surveyName}}</i> {{reportType}} link", {
surveyName: surveyName,
Expand Down
Loading