Add modality info and transforms for raw RGB, tokens, video descriptions, video transcripts, and video bounding boxes. #1
Conversation
|
Oops, I apologize for the extra files. Everythign except modality_transforms and modality_info.py should just be comments, so feel free to ignore them. |
fourm/data/modality_transforms.py
Outdated
| """ | ||
|
|
||
| # raise NotImplementedError("I'm not ") | ||
| def load(self, path): |
There was a problem hiding this comment.
Note: webd formats are not loaded using this function. This is only applicable for non-webd data storage formats (Ali).
There was a problem hiding this comment.
ml-4m/fourm/data/unified_datasets.py
Line 370 in 4c2c9a5
ml-4m/fourm/data/unified_datasets.py
Line 162 in 4c2c9a5
There was a problem hiding this comment.
All modalities should use webd and be compressed into webd/tar files.
There was a problem hiding this comment.
This means the load function will (probably) never be called for webd/our video-based modalities
| class VideoTokTransform(AbstractTransform): | ||
| """ | ||
| Assume input tokens is an ndarray of shape (num_frames, num_tokens_per_frame). | ||
| Transform the tokens to a torch tensor of shape (num_frames * num_tokens,). |
There was a problem hiding this comment.
Maybe don't flatten it here because this will make it a little bit more annoying to add the temporal embedding. Instead we can unflatten after adding the temporal embedding.
| def preprocess(self, sample): | ||
| return sample | ||
|
|
||
| def image_augment( |
There was a problem hiding this comment.
Role of this: do the augmentation during the tokenization step here
There was a problem hiding this comment.
TokTransform's v input is actually a list of tokenized inputs which are different augmentations of the same input. So that transform requires an index to select one of the augmented inputs.
There was a problem hiding this comment.
So here, we should assume no augmentations but keep the input v as a list of augmentations. but in this case since there's none just get v[0] always.
| def image_augment( | ||
| self, | ||
| bboxes_per_frame: List[List[Tuple]], | ||
| crop_coords: Tuple, |
There was a problem hiding this comment.
No rand_aug_idx for this because the other parameters actually specify which aug to apply
There was a problem hiding this comment.
so we have the same crop for each frame already!
| The innermost tuple is the actual representation of a bounding box instance (of the form [xmin, ymin, xmax, ymax, class_name, score]). | ||
| """ | ||
| instances = [frame["instances"] for frame in sample] | ||
| return self.convert_detection_instance(instances) |
There was a problem hiding this comment.
bugfix: wrap this in a list
| for i, bboxes in enumerate(bboxes_per_frame): | ||
| bboxes_str = self.convert_bboxes_to_string(bboxes_per_frame) | ||
| output_str += bboxes_str | ||
| output_str += f"<frame_{i+1}_token>" |
There was a problem hiding this comment.
these new special tokens need to be added to the tokenizer jsons -- https://github.com/swiss-ai/ml-4m/blob/main/fourm/utils/tokenizer/trained/text_tokenizer_4m_wordpiece_30k.json (we can manually add these tokens to the json file)
| bboxes_str = self.convert_bboxes_to_string(bboxes_per_frame) | ||
| output_str += bboxes_str | ||
| output_str += f"<frame_{i+1}_token>" | ||
| output_str += "<eos_token>" # TODO: Do we need to explicitly add EOS token here? If yes, we need to do this so it adds the actual eos_token str of the model not just this hardcoded string. |
There was a problem hiding this comment.
let's reuse the eos token from the text tokenizer
There was a problem hiding this comment.
| print( | ||
| "WARNING: no augmentations implemented for transcripts yet. Decide whether to augment/what these should be and then remove this warning." | ||
| ) | ||
| return val |
There was a problem hiding this comment.
No augmentations for now. val is actually the caption.
| def preprocess(self, sample): | ||
| return sample | ||
|
|
||
| def image_augment( |
There was a problem hiding this comment.
Precondition: assume val does not have any augmentations (thus list[dict])
postcondition: output is type list[dict]
…ionTransform, VideoTranscriptTransform, VideoDetTransform. Added a stub for videoRGBTransform. also sneak in a precommit
… keys and their mappings to transforms
Goal: Inputs from each modality, after being loaded from their paths, need to be transformed so that they can be used downstream for the model. These transforms are defined in
modality_transforms.py.modality_info.MODALITY_TRANSFORMSdefines the mapping from different modality IDs (e.g.,"tok_rgb") to its correspnding transformation. In this PR, we need to define new transformations for each video modality according to be represented as we have described in https://docs.google.com/presentation/d/1AY3QV1N_hoi9aXI1r8QTqrNmDK9LyorgJDQMPWb8hBo/edit#slide=id.g2e696416940_0_144.Summary: So far, I've implemented but NOT tested (like at all, because I haven't done any of the data setup yet :/ this is entirely just me spitballing on what the code should maybe look like and probably has a million little runtime errors, sorry :(( )
(a) the modality info mapping (I feel pretty good about this, but @garjania please take a look to see if that looks okay!) and
(b) tentative implementations for transforms for VideoTokTransform, VideoDescriptionTransform, VideoTranscriptTransform, and VideoDetTransform. Of these, I feel best about VideoDetTransform and VideoTokTransform. I have several TODOs in the code indicating questions. Most of these revolve around what kind of data augmentations (
image_augmentfunction) we should keep for the different video transforms. Also the input format and desired output shapes for some of these are not entirely clear to me (i.e., should the outputs already be unrolled from(num_frames, num_tokens_per_frame)to(num_frames * num_tokens_per_frame,)?), so any input there would be great!Remaining TODOs: Implement the raw RGB, clarify/confirm the questions, update the implementation accordingly, implement runtime fixes and test to make sure the transforms can all run correctly.
Reviewers: @garjania @vesteinn @yahya