Merge pull request #94 from out-of-cheese-error/feat/more_filters

Feat/more filters

CHANGELOG.md

@@ -8,7 +8,9 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## Unreleased
 ### Added
 - Nested tag support - "parent/child" tags used with the "Tag" hierarchy create nested folders.
-- Separate make and index commands, allow filtering annotations in both (Issue [#90](https://github.com/out-of-cheese-error/gooseberry/issues/90))
+- Separate make and index commands, allow filtering annotations in both (
+  Issue [#90](https://github.com/out-of-cheese-error/gooseberry/issues/90))
+- Better and more filtering options (Issue [#92](https://github.com/out-of-cheese-error/gooseberry/issues/92))
 
 ## [0.8.1] - 2021-03-14
 ### Changed

README.md

@@ -27,6 +27,7 @@ This demonstrates the interactive search functionality. `Enter` adds a new tag,
 * [Motivation](#motivation)
 * [A typical workflow](#a-typical-workflow)
 * [Some advantages](#some-advantages)
+* [Filtering](#filtering)
 * [Customization](#customization)
     * [Hypothesis](#hypothesis)
     * [Knowledge base](#knowledge-base)
@@ -124,18 +125,71 @@ you got it from, if ever you feel like you're missing context.
 * If you're in the mood, the note-taking won't involve window switching.
 * Even without using the wiki functionality you end up with a CLI to quickly tag your Hypothesis annotations.
 * Even without using the tagging functionality you end up with a pretty cool wiki listing all your annotations.
-* Since it's just plaintext, and the template can be customized, you can integrate it with any knowledge base system accepting plaintext files
+* Since it's just plaintext, and the template can be customized, you can integrate it with any knowledge base system
+  accepting plaintext files
   (like Obsidian, mdBook, org-mode, vim-wiki, etc.)
 
+## Filtering
+
+You can filter the annotations you want to modify or export using the following options in most gooseberry commands:
+
+```
+FLAGS:
+    -i, --include-updated
+            Include annotations updated in given time range (instead of just created)
+
+    -n, --not
+            Annotations NOT matching the given filter criteria
+
+    -o, --or
+            (Use with --tags) Annotations matching ANY of the given tags
+
+    -p, --page
+            Only page notes
+
+    -a, --annotation
+            Only annotations (i.e exclude page notes)
+
+
+OPTIONS:
+        --from <from>
+            Only annotations created after this date and time
+
+            Can be colloquial, e.g. "last Friday 8pm"
+        --before <before>
+            Only annotations created before this date and time
+
+            Can be colloquial, e.g. "last Friday 8pm"
+        --uri <uri>
+            Only annotations with this pattern in their URL
+
+            Doesn't have to be the full URL, e.g. "wikipedia" [default: ]
+        --any <any>
+            Only annotations with this pattern in their `quote`, `tags`, `text`, or `uri` [default: ]
+
+        --tags <tags>...
+            Only annotations with ALL of these tags (use --or to match ANY)
+
+        --exclude-tags <exclude-tags>...
+            Only annotations without ANY of these tags
+
+        --quote <quote>
+            Only annotations that contain this text inside the text that was annotated [default: ]
+
+        --text <text>
+            Only annotations that contain this text in their textual body [default: ]
+```
+
 ## Customization
 
 The default config TOML file is located in
 
 * Linux: `/home/<username>/.config`
 * Mac: `/Users/<username>/Library/Preferences`
 
-Change this by creating a config file with `gooseberry config default > config.toml` and modifying the contents. You can then use this as your
-configuration with `gooseberry -c path/to/config.toml <subcommand>` or by setting the environment variable `$GOOSEBERRY_CONFIG` to point to the file.
+Change this by creating a config file with `gooseberry config default > config.toml` and modifying the contents. You can
+then use this as your configuration with `gooseberry -c path/to/config.toml <subcommand>` or by setting the environment
+variable `$GOOSEBERRY_CONFIG` to point to the file.
 
 ### Hypothesis
 
@@ -146,12 +200,13 @@ Gooseberry takes annotations from a given Hypothesis group which you can create/
 
 ### Knowledge base
 
-You can set all the below options at once by running `gooseberry config kb all` or changing the corresponding keys in the config file (found
-at `gooseberry config location`)
+You can set all the below options at once by running `gooseberry config kb all` or changing the corresponding keys in
+the config file (found at `gooseberry config location`)
 
-Generate knowledge base files using `gooseberry make` - this command has options to filter annotations, and to clear the directory before generating (`-c` or `--clear`).
-By default, it also generates an index file (configured by the `index` and `link` configuration options below) using the filtered annotations - this can be disabled with `-n` or `--no-index`.
-Use `gooseberry index` to generate just the index file, this command also has annotation filtering options. 
+Generate knowledge base files using `gooseberry make` - this command has options to filter annotations, and to clear the
+directory before generating (`-c` or `--clear`). By default, it also generates an index file (configured by the `index`
+and `link` configuration options below) using the filtered annotations - this can be disabled with `--no-index`.
+Use `gooseberry index` to generate just the index file, this command also has annotation filtering options.
 
 #### Knowledge base directory
 

src/gooseberry/cli.rs

@@ -31,7 +31,8 @@ pub struct GooseberryCLI {
 pub enum GooseberrySubcommand {
     /// Sync newly added or updated Hypothesis annotations.
     Sync,
-    /// Opens a search buffer to see, filter, delete, add tags to and delete tags from annotations
+    /// Opens a search buffer to filter annotations.
+    /// Has keyboard shortcuts for deleting annotations, modifying tags, and creating knowledge-base files
     Search {
         #[structopt(flatten)]
         filters: Filters,
@@ -84,8 +85,8 @@ pub enum GooseberrySubcommand {
         /// Don't ask for confirmation before clearing
         #[structopt(short, long, requires = "clear")]
         force: bool,
-        /// Don't make index file
-        #[structopt(short, long)]
+        /// Don't make an index file
+        #[structopt(long)]
         no_index: bool,
     },
     /// Create an index file using hierarchy and optionally filtered annotations
@@ -130,7 +131,7 @@ pub enum GooseberrySubcommand {
 }
 
 /// CLI options for filtering annotations
-#[derive(StructOpt, Debug, Default)]
+#[derive(StructOpt, Debug, Default, Clone)]
 pub struct Filters {
     /// Only annotations created after this date and time
     ///
@@ -142,7 +143,7 @@ pub struct Filters {
     /// Can be colloquial, e.g. "last Friday 8pm"
     #[structopt(long, parse(try_from_str = utils::parse_datetime), conflicts_with = "from")]
     pub before: Option<DateTime<Utc>>,
-    /// If true, includes annotations updated after --from or before --before (instead of just created)
+    /// Include annotations updated in given time range (instead of just created)
     #[structopt(short, long)]
     pub include_updated: bool,
     /// Only annotations with this pattern in their URL
@@ -153,9 +154,30 @@ pub struct Filters {
     /// Only annotations with this pattern in their `quote`, `tags`, `text`, or `uri`
     #[structopt(default_value, long)]
     pub any: String,
-    /// Only annotations with these tags
-    #[structopt(long)]
+    /// Only annotations with ALL of these tags (use --or to match ANY)
+    #[structopt(long, use_delimiter = true, multiple = true)]
     pub tags: Vec<String>,
+    /// Only annotations without ANY of these tags
+    #[structopt(long, use_delimiter = true, multiple = true)]
+    pub exclude_tags: Vec<String>,
+    /// Only annotations that contain this text inside the text that was annotated.
+    #[structopt(default_value, long)]
+    pub quote: String,
+    /// Only annotations that contain this text in their textual body.
+    #[structopt(default_value, long)]
+    pub text: String,
+    /// Annotations NOT matching the given filter criteria
+    #[structopt(short, long)]
+    pub not: bool,
+    /// (Use with --tags) Annotations matching ANY of the given tags
+    #[structopt(short, long, requires = "tags")]
+    pub or: bool,
+    /// Only page notes
+    #[structopt(short, long)]
+    pub page: bool,
+    /// Only annotations (i.e exclude page notes)
+    #[structopt(short, long, conflicts_with = "page")]
+    pub annotation: bool,
 }
 
 impl From<Filters> for SearchQuery {
@@ -180,6 +202,8 @@ impl From<Filters> for SearchQuery {
             } else {
                 Sort::Created
             },
+            quote: filters.quote,
+            text: filters.text,
             ..SearchQuery::default()
         }
     }

src/gooseberry/knowledge_base.rs

@@ -19,7 +19,6 @@ use crate::configuration::{
     OrderBy, DEFAULT_ANNOTATION_TEMPLATE, DEFAULT_INDEX_LINK_TEMPLATE, DEFAULT_PAGE_TEMPLATE,
 };
 use crate::errors::Apologize;
-use crate::gooseberry::cli::Filters;
 use crate::gooseberry::Gooseberry;
 use crate::utils;
 use crate::utils::{clean_uri, uri_to_filename};
@@ -269,7 +268,7 @@ impl Gooseberry {
     /// Make mdBook wiki
     pub async fn make(
         &mut self,
-        filters: Filters,
+        annotations: Vec<Annotation>,
         clear: bool,
         force: bool,
         make: bool,
@@ -288,19 +287,22 @@ impl Gooseberry {
             fs::remove_dir_all(&kb_dir)?;
             fs::create_dir_all(&kb_dir)?;
         }
-        self.make_book(filters, &kb_dir, make, index).await?;
+        self.make_book(annotations, &kb_dir, make, index).await?;
         Ok(())
     }
 
     /// Write markdown files for wiki
     async fn make_book(
         &self,
-        filters: Filters,
+        annotations: Vec<Annotation>,
         src_dir: &Path,
         make: bool,
         index: bool,
     ) -> color_eyre::Result<()> {
-        let pb = utils::get_spinner("Fetching annotations...");
+        let mut annotations = annotations
+            .into_iter()
+            .map(AnnotationTemplate::from_annotation)
+            .collect();
         let extension = self.config.file_extension.as_ref().unwrap();
         let index_file = src_dir.join(format!(
             "{}.{}",
@@ -314,24 +316,6 @@ impl Gooseberry {
 
         // Register templates
         let hbs = self.get_handlebars()?;
-
-        // Get all annotations
-        let mut annotations: Vec<_> = self
-            .filter_annotations(filters, None)
-            .await?
-            .into_iter()
-            .filter(|a| {
-                !a.tags.iter().any(|t| {
-                    self.config
-                        .ignore_tags
-                        .as_ref()
-                        .map(|ignore_tags| ignore_tags.contains(t))
-                        .unwrap_or(false)
-                })
-            })
-            .map(AnnotationTemplate::from_annotation)
-            .collect();
-        pb.finish_with_message(&format!("Fetched {} annotations", annotations.len()));
         let pb = utils::get_spinner("Building knowledge base...");
         sort_annotations(
             self.config.sort.as_ref().unwrap_or(&vec![OrderBy::Created]),

src/gooseberry/mod.rs

@@ -10,6 +10,7 @@ use crate::configuration::GooseberryConfig;
 use crate::errors::Apologize;
 use crate::gooseberry::cli::{ConfigCommand, Filters, GooseberryCLI, GooseberrySubcommand};
 use crate::gooseberry::knowledge_base::AnnotationTemplate;
+use crate::utils;
 
 /// Command-line interface with `structopt`
 pub mod cli;
@@ -103,9 +104,25 @@ impl Gooseberry {
                 clear,
                 force,
                 no_index,
-            } => self.make(filters, clear, force, true, !no_index).await,
+            } => {
+                self.make(
+                    self.filter_annotations_make(filters).await?,
+                    clear,
+                    force,
+                    true,
+                    !no_index,
+                )
+                .await
+            }
             GooseberrySubcommand::Index { filters } => {
-                self.make(filters, false, false, false, true).await
+                self.make(
+                    self.filter_annotations_make(filters).await?,
+                    false,
+                    false,
+                    false,
+                    true,
+                )
+                .await
             }
             GooseberrySubcommand::Clear { force } => self.clear(force),
             GooseberrySubcommand::Uri { filters, ids } => {
@@ -198,23 +215,78 @@ impl Gooseberry {
         filters: Filters,
         group: Option<String>,
     ) -> color_eyre::Result<Vec<Annotation>> {
-        let mut query: SearchQuery = filters.into();
-        query.user = self.api.user.0.to_owned();
-        query.group = match group {
+        let group = match group {
             Some(group) => group,
             None => self
                 .config
                 .hypothesis_group
                 .clone()
                 .expect("This should have been set by Config"),
         };
-        let mut annotations: Vec<_> = self
-            .api
-            .search_annotations_return_all(&mut query)
+        let mut query: SearchQuery = filters.clone().into();
+        query.user = self.api.user.0.to_owned();
+        query.group = group.to_string();
+        let mut annotations = if filters.or && !filters.tags.is_empty() {
+            let mut annotations = Vec::new();
+            for tag in &filters.tags {
+                let mut tag_query = query.clone();
+                tag_query.tags = vec![tag.to_string()];
+                annotations.extend(
+                    self.api
+                        .search_annotations_return_all(&mut tag_query)
+                        .await?,
+                );
+            }
+            annotations
+        } else {
+            self.api.search_annotations_return_all(&mut query).await?
+        };
+        if !filters.exclude_tags.is_empty() {
+            annotations.retain(|a| !a.tags.iter().any(|t| filters.exclude_tags.contains(t)));
+        }
+        if filters.page {
+            annotations.retain(|a| a.target.iter().all(|t| t.selector.is_empty()));
+        }
+        if filters.annotation {
+            annotations.retain(|a| a.target.iter().any(|t| !t.selector.is_empty()));
+        }
+        if filters.not {
+            let mut query: SearchQuery = Filters::default().into();
+            query.user = self.api.user.0.to_owned();
+            query.group = group;
+            let mut all_annotations: Vec<_> =
+                self.api.search_annotations_return_all(&mut query).await?;
+            let remove_ids = annotations.iter().map(|a| &a.id).collect::<HashSet<_>>();
+            all_annotations.retain(|a| !remove_ids.contains(&a.id));
+            annotations = all_annotations;
+        }
+        annotations.sort_by(|a, b| a.created.cmp(&b.created));
+        Ok(annotations)
+    }
+
+    /// Fetch annotations for knowledge base
+    /// Ignores annotations with tags in `ignore_tags` configuration option.
+    pub async fn filter_annotations_make(
+        &self,
+        filters: Filters,
+    ) -> color_eyre::Result<Vec<Annotation>> {
+        let pb = utils::get_spinner("Fetching annotations...");
+        // Get all annotations
+        let annotations: Vec<_> = self
+            .filter_annotations(filters, None)
             .await?
             .into_iter()
+            .filter(|a| {
+                !a.tags.iter().any(|t| {
+                    self.config
+                        .ignore_tags
+                        .as_ref()
+                        .map(|ignore_tags| ignore_tags.contains(t))
+                        .unwrap_or(false)
+                })
+            })
             .collect();
-        annotations.sort_by(|a, b| a.created.cmp(&b.created));
+        pb.finish_with_message(&format!("Fetched {} annotations", annotations.len()));
         Ok(annotations)
     }