Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

612 Introduced RECAP Search Alerts sweep index #4127

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
3e4f269
fix(elasticsearch): Test RECAP nested index reliability
albertisfu Jun 21, 2024
53b3b65
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jun 21, 2024
2955b0b
fix(alerts): Changed sweep index approach to parent-child documents
albertisfu Jun 22, 2024
9307b77
fix(alerts): Added cl_send_recap_alerts command
albertisfu Jun 25, 2024
9b4e1c1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 25, 2024
8b537f0
fix(alerts): Implemented filtering of RECAP alerts hits for the sweep…
albertisfu Jun 27, 2024
c1232ec
fix(alerts): Updated ES alert email templates to support RECAP Alerts.
albertisfu Jun 28, 2024
3e96f61
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jun 28, 2024
51c7bb6
fix(alerts): Group alerts and case hits limit
albertisfu Jun 28, 2024
7fc3298
fix(alerts): Trigger RECAP search alerts webhooks
albertisfu Jun 29, 2024
b5016ba
fix(alerts): Schedule wly and mly RECAP Search Alerts
albertisfu Jun 29, 2024
4a128bf
fix(alerts): Copy documents from the main index to the sweep index us…
albertisfu Jul 2, 2024
3a4a456
fix(alerts): Fixed RECAPSweepDocument index mapping
albertisfu Jul 2, 2024
add980a
fix(alerts): Tweak RECAP Alert estimation query to consider both Dock…
albertisfu Jul 3, 2024
a20113f
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jul 3, 2024
ebf269d
fix(elasticsearch): Fixed build_daterange_query type hint
albertisfu Jul 3, 2024
bffee6d
fix(alerts): Fixed re_index task estimated remaining time compute
albertisfu Jul 3, 2024
847f0fd
fix(alerts): Handle creation and removal of the RECAP alerts sweep in…
albertisfu Jul 3, 2024
4b324c9
fix(elasticsearch): Fixed tests related to timestamp updates
albertisfu Jul 3, 2024
0d63080
fix(alerts): Fix should_docket_hit_be_included date comparison
albertisfu Jul 4, 2024
5b3d130
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jul 4, 2024
5077e01
fix(alerts): Changed approach to filter out cross-object hits by usin…
albertisfu Jul 10, 2024
9dffbfd
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jul 10, 2024
a4e4e62
fix(alerts): Added more tests related to filtering cross-object hits.
albertisfu Jul 10, 2024
49dd480
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jul 10, 2024
a468336
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jul 19, 2024
38d6884
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Jul 25, 2024
b56f235
fix(alerts): Restore send_es_search_alert_webhook to avoid conflicts …
albertisfu Jul 25, 2024
d102664
fix(alerts): Fixed MLY alerts test can't be sent after the 28th
albertisfu Jul 29, 2024
7977b80
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Sep 26, 2024
57b6df7
fix(alerts): Fixed merge conflicts and adjust test accordingly new RE…
albertisfu Sep 26, 2024
b35ef0a
fix(elasticsearch): Fixed failing test due to build_full_join_es_quer…
albertisfu Sep 27, 2024
8902aa0
fix(alerts): Removed recap_document_hl_matched as we no longer rely o…
albertisfu Sep 27, 2024
d0b1298
Merge branch 'main' into 612-introduced-recap-search-alerts
albertisfu Sep 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
733 changes: 733 additions & 0 deletions cl/alerts/management/commands/cl_send_recap_alerts.py

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion cl/alerts/management/commands/cl_send_scheduled_alerts.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,9 @@ def query_and_send_alerts_by_rate(rate: str) -> None:
)
)
if hits:
send_search_alert_emails.delay([(user_id, hits)])
send_search_alert_emails.delay(
[(user_id, hits)], scheduled_alert=True
)
alerts_sent_count += 1

# Update Alert's date_last_hit in bulk.
Expand Down
2 changes: 1 addition & 1 deletion cl/alerts/management/commands/clean_up_search_alerts.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def validate_queries_syntax(options: OptionsType) -> None:
if search_form.is_valid():
cd = search_form.cleaned_data
try:
s, _ = build_es_base_query(search_query, cd)
s, _, _ = build_es_base_query(search_query, cd)
s = s.extra(size=0)
s.execute().to_dict()
# Waiting between requests to avoid hammering ES too quickly.
Expand Down
12 changes: 8 additions & 4 deletions cl/alerts/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
from cl.api.models import WebhookEventType
from cl.api.tasks import (
send_docket_alert_webhook_events,
send_es_search_alert_webhook,
send_search_alert_webhook_es,
)
from cl.celery_init import app
from cl.custom_filters.templatetags.text_filters import best_case_name
Expand Down Expand Up @@ -458,22 +458,25 @@ def send_webhook_alert_hits(
event_type=WebhookEventType.SEARCH_ALERT, enabled=True
)
for user_webhook in user_webhooks:
send_es_search_alert_webhook.delay(
send_search_alert_webhook_es.delay(
documents,
user_webhook.pk,
alert,
alert.pk,
)


@app.task(ignore_result=True)
def send_search_alert_emails(
email_alerts_to_send: list[tuple[int, list[SearchAlertHitType]]]
email_alerts_to_send: list[tuple[int, list[SearchAlertHitType]]],
scheduled_alert: bool = False,
) -> None:
"""Send search alert emails for multiple users.

:param email_alerts_to_send: A list of two tuples containing the user to
whom the alerts should be sent. A list of tuples containing the Search
Alert, (Alert, search type, documents, and number of documents)
:param scheduled_alert: A boolean indicating weather this alert has been
scheduled
:return: None
"""

Expand All @@ -491,6 +494,7 @@ def send_search_alert_emails(
context = {
"hits": hits,
"hits_limit": settings.SCHEDULED_ALERT_HITS_LIMIT,
"scheduled_alert": scheduled_alert,
}
headers = {}
query_string = ""
Expand Down
72 changes: 55 additions & 17 deletions cl/alerts/templates/alert_email_es.html
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ <h1 class="bottom" style="font-size: 3em; font-weight: normal; line-height: 1;
{% for result in results %}
{% if forloop.first %}
<h2 style="font-size: 2em; font-weight: normal; font-family: inherit; color: #111; border: 0; vertical-align: baseline; font-style: inherit; margin: 0; padding: 0;" class="alt bottom">
Your {{alert.get_rate_display|lower}} {% if type == 'o' %}opinion{% elif type == 'oa' %}oral argument{% endif %} alert &mdash; {{alert.name}} &mdash; had {{num_results}}{% if num_results >= hits_limit %}+{% endif %} hit{{results|pluralize}}:
Your {{alert.get_rate_display|lower}} {% if type == 'o' %}opinion{% elif type == 'oa' %}oral argument{% elif type == 'r' %}RECAP{% endif %} alert &mdash; {{alert.name}} &mdash; had {{num_results}}{% if num_results >= hits_limit %}+{% endif %} hit{{results|pluralize}}:
</h2>
<p style="font-size: 100%; font-weight: inherit; font-family: inherit; border: 0; vertical-align: baseline; font-style: inherit; margin: 0 0 1.5em; padding: 0;">
<a href="https://www.courtlistener.com/?{{ alert.query_run|safe }}&edit_alert={{ alert.pk }}">View Full Results / Edit this Alert</a><br>
Expand All @@ -34,28 +34,66 @@ <h2 style="font-size: 2em; font-weight: normal; font-family: inherit; color: #11
{% endif %}

<h3 class="alt bottom" style="font-size: 1.5em; font-weight: normal; line-height: 1; font-family: 'Warnock Pro', 'Goudy Old Style','Palatino','Book Antiqua', Georgia, serif; color: #666; border: 0; vertical-align: baseline; font-style: italic; margin: 0; padding: 0;">
<a href="https://www.courtlistener.com{{result.absolute_url}}" style="font-size: 100%; font-weight: inherit; font-family: inherit; color: #009; border: 0; font-style: inherit; padding: 0; text-decoration: none; vertical-align: baseline; margin: 0;">
<a href="https://www.courtlistener.com{% if type == 'r' %}{{result.docket_absolute_url}}{% else %}{{result.absolute_url}}{% endif %}" style="font-size: 100%; font-weight: inherit; font-family: inherit; color: #009; border: 0; font-style: inherit; padding: 0; text-decoration: none; vertical-align: baseline; margin: 0;">
{{ forloop.counter }}. {{ result|get_highlight:"caseName"|safe }}
({% if result.court_id != 'scotus' %}{{ result|get_highlight:"court_citation_string"|nbsp|safe }}&nbsp;{% endif %}{% if type == 'o' %}{{ result.dateFiled|date:"Y" }}{% elif type == 'oa' %}{{ result.dateArgued|date:"Y" }}{% endif %})
({% if result.court_id != 'scotus' %}{{ result|get_highlight:"court_citation_string"|nbsp|safe }}&nbsp;{% endif %}{% if type == 'o' or type == 'r' %}{{ result.dateFiled|date:"Y" }}{% elif type == 'oa' %}{{ result.dateArgued|date:"Y" }}{% endif %})
</a>
</h3>
<p style="font-size: 100%; font-weight: inherit; font-family: inherit; border: 0; vertical-align: baseline; font-style: inherit; margin: 0; padding: 0;">
<strong style="font-weight: bold;">
View original:
</strong>
{% if result.download_url %}
<a href="{{result.download_url}}" style="font-size: 100%; font-weight: inherit; font-family: inherit; color: #009; border: 0; font-style: inherit; padding: 0; text-decoration: none; vertical-align: baseline; margin: 0;">
From the court
{% if type == 'r' %}
{% if result.docketNumber %}
<strong style="font-weight: bold;">Docket Number: </strong>
{{ result|get_highlight:"docketNumber"|safe }}
{% endif %}
<br>
<strong style="font-weight: bold;">Date Filed: </strong>
{% if result.dateFiled %}
{{ result.dateFiled|date:"F jS, Y" }}
{% else %}
Unknown Date
{% endif %}
<ul>
{% for doc in result.child_docs %}
{% with doc=doc|get_es_doc_content:scheduled_alert %}
<li>
<a href="https://www.courtlistener.com{% if doc.absolute_url %}{{ doc.absolute_url }}{% else %}{{ result.docket_absolute_url }}#minute-entry-{{ doc.docket_entry_id }}{% endif %}" class="visitable">{% if doc.short_description %}{{ doc.short_description|render_string_or_list|safe }}<span class="gray">&nbsp;&mdash;&nbsp;</span>{% endif %}Document #{% if doc.document_number %}{{ doc.document_number }}{% endif %}{% if doc.attachment_number %}, Attachment #{{ doc.attachment_number }}{% endif %}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detected a segment of a Flask template where autoescaping is explicitly disabled with '| safe' filter. This allows rendering of raw HTML in this segment. Ensure no user data is rendered here, otherwise this is a cross-site scripting (XSS) vulnerability.

Ignore this finding from template-unescaped-with-safe.

</a>
{% if doc.description %}
<span style="display: block; margin-top: 5px;">Description: {{ doc.description|render_string_or_list|safe }}</span>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detected a segment of a Flask template where autoescaping is explicitly disabled with '| safe' filter. This allows rendering of raw HTML in this segment. Ensure no user data is rendered here, otherwise this is a cross-site scripting (XSS) vulnerability.

Ignore this finding from template-unescaped-with-safe.

{% endif %}
{% if doc.plain_text %}
{% contains_highlights doc.plain_text.0 True as highlighted %}
<span style="display: block; margin-top: 5px;">{% if highlighted %}&hellip; {% endif %}{{ doc.plain_text|render_string_or_list|safe|underscore_to_space }} &hellip;</span>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detected a segment of a Flask template where autoescaping is explicitly disabled with '| safe' filter. This allows rendering of raw HTML in this segment. Ensure no user data is rendered here, otherwise this is a cross-site scripting (XSS) vulnerability.

Ignore this finding from template-unescaped-with-safe.

{% endif %}
</li>
{% endwith %}
{% endfor %}
</ul>
{% if result.child_docs and result.child_remaining %}
{% extract_q_value alert.query_run as q_value %}
<a href="https://www.courtlistener.com/?type={{ type|urlencode }}&q={% if q_value %}({{ q_value|urlencode }})%20AND%20{% endif %}docket_id%3A{{ result.docket_id|urlencode }}">
<strong>View Additional Results for this Case</strong>
</a>
<br>
{% endif %}
{% else %}
<p style="font-size: 100%; font-weight: inherit; font-family: inherit; border: 0; vertical-align: baseline; font-style: inherit; margin: 0; padding: 0;">
<strong style="font-weight: bold;">
View original:
</strong>
{% if result.download_url %}
<a href="{{result.download_url}}" style="font-size: 100%; font-weight: inherit; font-family: inherit; color: #009; border: 0; font-style: inherit; padding: 0; text-decoration: none; vertical-align: baseline; margin: 0;">
From the court
</a>
&nbsp;&nbsp;|&nbsp;&nbsp;
{% endif %}
{% if result.local_path %}
{# Provide link to S3. #}
<a href="https://storage.courtlistener.com/{{result.local_path}}" style="font-size: 100%; font-weight: inherit; font-family: inherit; color: #009; border: 0; font-style: inherit; padding: 0; text-decoration: none; vertical-align: baseline; margin: 0;">
Our backup
</a>
&nbsp;&nbsp;|&nbsp;&nbsp;
{% endif %}
{% if result.local_path %}
{# Provide link to S3. #}
<a href="https://storage.courtlistener.com/{{result.local_path}}" style="font-size: 100%; font-weight: inherit; font-family: inherit; color: #009; border: 0; font-style: inherit; padding: 0; text-decoration: none; vertical-align: baseline; margin: 0;">
Our backup
</a>
</p>
{% endif %}
</p>
{% if type == 'oa' %}
<p style="font-size: 100%; font-weight: inherit; font-family: inherit; border: 0; vertical-align: baseline; font-style: inherit; margin: 0 0 0; padding: 0;">
<strong style="font-weight: bold;">Date Argued: </strong>
Expand Down
15 changes: 11 additions & 4 deletions cl/alerts/templates/alert_email_es.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,23 @@ CourtListener.com
We have news regarding your alerts at CourtListener.com
-------------------------------------------------------

{% for alert, type, results, num_results in hits %}{% for result in results %}{% if forloop.first %}Your {{alert.get_rate_display|lower}} {% if type == 'o' %}opinion{% elif type == 'oa' %}oral argument{% endif %} alert -- {{alert.name}} -- had {{num_results}}{% if num_results >= hits_limit %}+{% endif %} hit{{results|pluralize}}:
{% for alert, type, results, num_results in hits %}{% for result in results %}{% if forloop.first %}Your {{alert.get_rate_display|lower}} {% if type == 'o' %}opinion{% elif type == 'oa' %}oral argument{% elif type == 'r' %}RECAP{% endif %} alert -- {{alert.name}} -- had {{num_results}}{% if num_results >= hits_limit %}+{% endif %} hit{{results|pluralize}}:
-------------------------------------------------------
View Full Results / Edit this Alert: https://www.courtlistener.com/?{{ alert.query_run|safe }}&edit_alert={{ alert.pk }}
Disable this Alert (one click): https://www.courtlistener.com{% url "disable_alert" alert.secret_key %}{% endif %}

{{forloop.counter}}. {{ result.caseName|render_string_or_list|safe|striptags }} ({% if result.court_id != 'scotus' %}{{ result.court_citation_string|render_string_or_list|striptags }} {% endif %}{% if type == 'o' %}{{ result.dateFiled|date:"Y" }}{% elif type == 'oa' %}{{ result.dateArgued|date:"Y" }}{% endif %})
{{forloop.counter}}. {{ result.caseName|render_string_or_list|safe|striptags }} ({% if result.court_id != 'scotus' %}{{ result.court_citation_string|render_string_or_list|striptags }} {% endif %}{% if type == 'o' or type == 'r' %}{{ result.dateFiled|date:"Y" }}{% elif type == 'oa' %}{{ result.dateArgued|date:"Y" }}{% endif %})
{% if type == 'oa' %}{% if result.dateArgued %}Date Argued: {{ result.dateArgued|date:"F jS, Y" }}{% else %}Date Argued: Unknown Date {% endif %}{% if result.docketNumber %} | Docket Number: {{ result.docketNumber|render_string_or_list|safe|striptags }}{% endif %} | Duration: {{ result.duration|naturalduration }}{% if result.judge %} | Judge: {{ result.judge|render_string_or_list|safe|striptags|underscore_to_space }}{% endif %}{% endif %}
{% if type == 'o' or type == 'oa' %}{% if result|get_highlight:"text" %}...{{ result|get_highlight:"text"|safe|striptags|underscore_to_space|compress_whitespace }}...{% endif %}
{% if type == 'o' or type == 'oa' %}{% if result|get_highlight:"text" %}...{{ result|get_highlight:"text"|safe|striptags|underscore_to_space|compress_whitespace }}...{% endif %}{% endif %}
{% if type == 'r' %}{% if result.dateFiled %}Date Filed: {{ result.dateFiled|date:"F jS, Y" }}{% else %}Date Filed: Unknown Date {% endif %}{% if result.docketNumber %} | Docket Number: {{ result.docketNumber|render_string_or_list|safe|striptags }}{% endif %}
{% for doc in result.child_docs %}{% with doc=doc|get_es_doc_content:scheduled_alert %} - {% if doc.short_description %}{{ doc.short_description|render_string_or_list|safe|striptags }} - {% endif %}Document #{% if doc.document_number %}{{ doc.document_number }}{% endif %}{% if doc.attachment_number %}, Attachment #{{ doc.attachment_number }}{% endif %}
{% if doc.description %}Description: {{ doc.description|render_string_or_list|safe|striptags }}{% endif %}
{% if doc.plain_text %}{% contains_highlights doc.plain_text.0 True as highlighted %}{% if highlighted %}...{% endif %}{{ doc.plain_text|render_string_or_list|safe|striptags|underscore_to_space }}...{% endif %}
View this document on our site: https://www.courtlistener.com{% if doc.absolute_url %}{{ doc.absolute_url }}{% else %}{{ result.docket_absolute_url }}#minute-entry-{{ doc.docket_entry_id }}{% endif %}
{% endwith %}{% endfor %}
{% if result.child_docs and result.child_remaining %}{% extract_q_value alert.query_run as q_value %}View Additional Results for this Case: https://www.courtlistener.com/?type={{ type|urlencode }}&q={% if q_value %}({{ q_value|urlencode }})%20AND%20{% endif %}docket_id%3A{{ result.docket_id|urlencode }}{% endif %}
{% endif %}~~~~~
- View this item on our site: https://www.courtlistener.com{{result.absolute_url}}
- View this item on our site: https://www.courtlistener.com{% if type == 'r' %}{{result.docket_absolute_url}}{% else %}{{result.absolute_url}}{% endif %}
{% if result.download_url %} - Download original from the court: {{result.download_url}}
{% endif %}{% if result.local_path %} - Download the original from our backup: https://storage.courtlistener.com/{{ result.local_path }}{% endif %}{% endfor %}

Expand Down
Empty file added cl/alerts/tests/__init__.py
Empty file.
File renamed without changes.
Loading
Loading