tfrecord-beam-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1df6c870-747a-44c2-854e-bcd5ed14f320",
   "metadata": {},
   "source": [
    "## Beam conversion from Bigquery to TF Records\n",
    "\n",
    "In this notebook we use Apache Beam to convert to tfrecords\n",
    "The applications can be found in `beam_candidates` and `beam_training` for candidate generation and training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e0f365f-5d78-4114-b741-311ffcc64884",
   "metadata": {},
   "source": [
    "`pip install --upgrade 'apache-beam[gcp]'`\n",
    "\n",
    "#### IMPORTANT - make sure you upgrade Dataflow with the above command then restart the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1bab01ef-6f4f-47fe-8f54-56bae24cae25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dill==0.3.1.1\n"
     ]
    }
   ],
   "source": [
    "# !pip install --upgrade 'apache-beam[gcp]'\n",
    "!pip freeze | grep dill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3148ce99-d7d0-4298-a3d3-4ea5a1aee30b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1176b3bf-c98f-4d44-a410-31f816b1998d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! gsutil mb -l us-central1 gs://spotify-beam-v3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "880e9bb8-c550-414b-9b65-b2124837e937",
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup\n",
    "PROJECT_ID = 'hybrid-vertex'\n",
    "BUCKET_NAME = 'spotify-beam-v3' # 'spotify-tfrecords-blog' # Set your Bucket name\n",
    "REGION = 'us-west1' # Set the region for Dataflow jobs\n",
    "VERSION = 'v4'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6babbe19-0ea7-42cc-ac89-db2705f4ae16",
   "metadata": {},
   "source": [
    "### Run the Dataflow app to convert from BQ to TFrecords\n",
    "\n",
    "Candidate generation can be found in `beam_candidates`\n",
    "Training and Validation generation can be found in `beam_training`\n",
    "\n",
    "Usage:\n",
    "\n",
    "  Candidate generation \n",
    "  \n",
    "  `beam_candidates\\python3 main.py`\n",
    "   \n",
    "  Training generation\n",
    "  \n",
    "  `beam_training\\python3 main-train.py <BQ_table> <gcs data subfolder>`\n",
    "  \n",
    "  \n",
    "##### Be careful with quotas - running more than two jobs can run into quota issues with defaults\n",
    "\n",
    "Training data generation runs about 1 hour with 10 workers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5a82dfad-cdab-40eb-9859-ef892c8fa2df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jupyter/spotify-tfrs\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "81357053-6541-4fd2-8e04-b96474b797b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jupyter/spotify-tfrs/beam_training\n"
     ]
    }
   ],
   "source": [
    "%cd beam_training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3489703c-119d-4685-b0ad-9d13e7e14e1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jupyter/spotify-tfrs/beam_training\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af90abd7-675a-4a3c-8cec-ad78db3355cd",
   "metadata": {},
   "source": [
    "### Original train & valid sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "34ef7098-b83e-4263-aefd-56b83ee8155c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Expected TFRecords: 3404\n",
      "GoogleCloudOptions(create_from_snapshot=None, dataflow_endpoint=https://dataflow.googleapis.com, dataflow_kms_key=None, dataflow_service_options=None, enable_artifact_caching=False, enable_hot_key_logging=False, enable_streaming_engine=False, flexrs_goal=None, impersonate_service_account=None, job_name=spotify-bq-tfrecords-v5-221004-171739, labels=None, no_auth=False, project=hybrid-vertex, region=us-west1, service_account_email=None, staging_location=gs://spotify-beam-v3/v5/job/staging/, temp_location=gs://spotify-beam-v3/v5/job/temp/, template_location=None, transform_name_mapping=None, update=False)\n",
      "/opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py:2739: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported\n",
      "  temp_location = pcoll.pipeline.options.view_as(\n",
      "warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n",
      "\n",
      "warning: check: missing required meta-data: url\n",
      "\n",
      "warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! python3 main-train.py train_flatten train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e608a184-d7e1-4f93-a5df-bf33447b36fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Expected TFRecords: 3404\n",
      "GoogleCloudOptions(create_from_snapshot=None, dataflow_endpoint=https://dataflow.googleapis.com, dataflow_kms_key=None, dataflow_service_options=None, enable_artifact_caching=False, enable_hot_key_logging=False, enable_streaming_engine=False, flexrs_goal=None, impersonate_service_account=None, job_name=spotify-bq-tfrecords-v5-221004-165345, labels=None, no_auth=False, project=hybrid-vertex, region=us-west1, service_account_email=None, staging_location=gs://spotify-beam-v3/v5/job/staging/, temp_location=gs://spotify-beam-v3/v5/job/temp/, template_location=None, transform_name_mapping=None, update=False)\n",
      "/opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py:2739: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported\n",
      "  temp_location = pcoll.pipeline.options.view_as(\n",
      "warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n",
      "\n",
      "warning: check: missing required meta-data: url\n",
      "\n",
      "warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! python3 main-train.py train_flatten_valid valid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8febf249-baaf-4f88-a526-da047670b5d2",
   "metadata": {},
   "source": [
    "### Generating \"different artist\" tfrecords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1f4b63c-f8ab-4cc1-b0a5-17b110d102e9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Expected TFRecords: 3404\n",
      "GoogleCloudOptions(create_from_snapshot=None, dataflow_endpoint=https://dataflow.googleapis.com, dataflow_kms_key=None, dataflow_service_options=None, enable_artifact_caching=False, enable_hot_key_logging=False, enable_streaming_engine=False, flexrs_goal=None, impersonate_service_account=None, job_name=spotify-bq-tfrecords-v5-221004-141012, labels=None, no_auth=False, project=hybrid-vertex, region=us-west1, service_account_email=None, staging_location=gs://spotify-beam-v3/v5/job/staging/, temp_location=gs://spotify-beam-v3/v5/job/temp/, template_location=None, transform_name_mapping=None, update=False)\n",
      "/opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py:2739: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported\n",
      "  temp_location = pcoll.pipeline.options.view_as(\n",
      "warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n",
      "\n",
      "warning: check: missing required meta-data: url\n",
      "\n",
      "warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! python3 main-train.py train_flatten_dif_artist_valid dif_artist_valid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b99498ca-5f49-4c26-be8d-55046eca46be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Expected TFRecords: 3404\n",
      "GoogleCloudOptions(create_from_snapshot=None, dataflow_endpoint=https://dataflow.googleapis.com, dataflow_kms_key=None, dataflow_service_options=None, enable_artifact_caching=False, enable_hot_key_logging=False, enable_streaming_engine=False, flexrs_goal=None, impersonate_service_account=None, job_name=spotify-bq-tfrecords-v5-221004-143903, labels=None, no_auth=False, project=hybrid-vertex, region=us-west1, service_account_email=None, staging_location=gs://spotify-beam-v3/v5/job/staging/, temp_location=gs://spotify-beam-v3/v5/job/temp/, template_location=None, transform_name_mapping=None, update=False)\n",
      "/opt/conda/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py:2739: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported\n",
      "  temp_location = pcoll.pipeline.options.view_as(\n",
      "warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n",
      "\n",
      "warning: check: missing required meta-data: url\n",
      "\n",
      "warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! python3 main-train.py train_flatten_dif_artist dif_artist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a33f54-c717-4130-b375-3108836e9e04",
   "metadata": {},
   "source": [
    "## Test the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "5c7c9dd2-7b32-4cb9-a7f9-97a0b1cdd487",
   "metadata": {},
   "outputs": [],
   "source": [
    "## testing output\n",
    "from google.cloud import storage\n",
    "\n",
    "client = storage.Client()\n",
    "# # gs://spotify-beam-v3/v3/candidates/*.tfrecords\n",
    "\n",
    "BUCKET = 'spotify-beam-v3'\n",
    "CANDIDATE_PREFIX = 'v3/candidates/'\n",
    "\n",
    "candidate_files = []\n",
    "for blob in client.list_blobs(f\"{BUCKET}\", prefix=f'{CANDIDATE_PREFIX}', delimiter=\"/\"):\n",
    "    candidate_files.append(blob.public_url.replace(\"https://storage.googleapis.com/\", \"gs://\"))\n",
    "    \n",
    "candidate_dataset = tf.data.TFRecordDataset(candidate_files)\n",
    "\n",
    "def parse_candidate_tfrecord_fn(example):\n",
    "    example = tf.io.parse_single_example(\n",
    "        example, \n",
    "        features=candidate_features\n",
    "    )\n",
    "    return example\n",
    "\n",
    "# parsed_candidate_dataset = candidate_dataset.map(parse_candidate_tfrecord_fn, num_parallel_calls=-1)\n",
    "\n",
    "parsed_candidate_dataset = candidate_dataset.map(parse_candidate_tfrecord_fn) ### THIS NEEDS TO BE FIXED SO THE UNIQUE PRODUCT DATASET HAS THE SAME FIELD NAMES (goes thru the same model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "b57c12cb-6c95-4bac-ba29-4d389babc44c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'album_name_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'The Sound of Everything Rmx',\n",
      "       b'World Psychedelic Classics 4: Nobody Can Live Forever: The Existential Soul of Tim Maia'],\n",
      "      dtype=object)>, 'artist_followers_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([ 277649., 1363781.], dtype=float32)>, 'artist_genres_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b\"'downtempo', 'electronica', 'funk', 'latin alternative', 'nu jazz', 'nu-cumbia', 'trip hop', 'world'\",\n",
      "       b\"'brazilian boogie', 'brazilian soul', 'mpb', 'samba'\"],\n",
      "      dtype=object)>, 'artist_name_can': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'Quantic', b'Tim Maia'], dtype=object)>, 'artist_pop_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([64., 64.], dtype=float32)>, 'artist_uri_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:artist:5ZMwoAjeDtLJ0XRwRTgaK8',\n",
      "       b'spotify:artist:0jOs0wnXCu1bGGP7kh5uIu'], dtype=object)>, 'duration_ms_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([267130., 199720.], dtype=float32)>, 'track_pop_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([53.,  0.], dtype=float32)>, 'track_uri_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:track:27CDzo2P7Mf3dKoa76tNxb',\n",
      "       b'spotify:track:4Eub2uHpLjK4fY3qR9uX8U'], dtype=object)>}\n"
     ]
    }
   ],
   "source": [
    "for x in parsed_candidate_dataset.batch(2).take(1):\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "49445d15-3782-42e3-9820-8494ad67ac7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_features = {\n",
    "    'track_name_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'artist_name_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'album_name_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'track_uri_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'artist_uri_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'album_uri_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'duration_ms_can': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'track_pop_can': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'artist_pop_can': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'artist_genres_can': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'artist_followers_can': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    # 'pos_seed_track': tf.io.FixedLenFeature(dtype=tf.int64, shape=()),\n",
    "    'track_name_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'artist_name_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'album_name_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'track_uri_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'artist_uri_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'album_uri_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'duration_seed_track': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'track_pop_seed_track': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'artist_pop_seed_track': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'artist_genres_seed_track': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'artist_followers_seed_track': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    # 'pid': tf.io.FixedLenFeature(dtype=tf.int64, shape=()),\n",
    "    'name': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    'collaborative': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    # 'duration_ms_seed_pl': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'n_songs_pl': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'num_artists_pl': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'num_albums_pl': tf.io.FixedLenFeature(dtype=tf.float32, shape=()),\n",
    "    'description_pl': tf.io.FixedLenFeature(dtype=tf.string, shape=()),\n",
    "    ###ragged\n",
    "    'track_name_pl': tf.io.RaggedFeature(tf.string),\n",
    "    'artist_name_pl': tf.io.RaggedFeature(tf.string),\n",
    "    'album_name_pl': tf.io.RaggedFeature(tf.string),\n",
    "    'track_uri_pl': tf.io.RaggedFeature(tf.string),\n",
    "    'duration_ms_songs_pl': tf.io.RaggedFeature(tf.float32),\n",
    "    'artist_pop_pl': tf.io.RaggedFeature(tf.float32),\n",
    "    'artists_followers_pl': tf.io.RaggedFeature(tf.float32),\n",
    "    'track_pop_pl': tf.io.RaggedFeature(tf.float32),\n",
    "    'artist_genres_pl': tf.io.RaggedFeature(tf.string),\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "bdad96f5-0e70-427b-9525-081406b9c80f",
   "metadata": {},
   "outputs": [],
   "source": [
    "## testing output\n",
    "from google.cloud import storage\n",
    "\n",
    "client = storage.Client()\n",
    "# # gs://spotify-beam-v3/v3/candidates/*.tfrecords\n",
    "\n",
    "BUCKET = 'spotify-beam-v3'\n",
    "CANDIDATE_PREFIX = 'v3/valid/'\n",
    "\n",
    "valid_files = []\n",
    "for blob in client.list_blobs(f\"{BUCKET}\", prefix=f'{CANDIDATE_PREFIX}', delimiter=\"/\"):\n",
    "    valid_files.append(blob.public_url.replace(\"https://storage.googleapis.com/\", \"gs://\"))\n",
    "    \n",
    "valid = tf.data.TFRecordDataset(valid_files)\n",
    "\n",
    "def parse_tfrecord(example):\n",
    "    example = tf.io.parse_single_example(\n",
    "        example, \n",
    "        features=all_features\n",
    "    )\n",
    "    return example\n",
    "\n",
    "# parsed_candidate_dataset = candidate_dataset.map(parse_candidate_tfrecord_fn, num_parallel_calls=-1)\n",
    "\n",
    "valid_parsed = valid.map(parse_tfrecord) ### THIS NEEDS TO BE FIXED SO THE UNIQUE PRODUCT DATASET HAS THE SAME FIELD NAMES (goes thru the same model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "a6b87491-5706-4204-a19e-582612c1b439",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'album_name_can': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'The Big Revival', b'Metro Station'], dtype=object)>, 'album_name_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'Country Strong (Original Motion Picture Soundtrack)',\n",
      "       b'Nights Like These'], dtype=object)>, 'album_uri_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:album:2R5PAxygJ4YeRuSwoalKam',\n",
      "       b'spotify:album:7Eb0IrQS3xbuje3EsREVnb'], dtype=object)>, 'album_uri_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:album:0GqBF7BpSBxXSyRd5ouCrj',\n",
      "       b'spotify:album:67xqQNdhQSnYqAwOyYZHom'], dtype=object)>, 'artist_followers_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([3827446.,  293524.], dtype=float32)>, 'artist_followers_seed_track': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([2477411.,   64356.], dtype=float32)>, 'artist_genres_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b\"'contemporary country', 'country', 'country road'\",\n",
      "       b\"'electropowerpop', 'neon pop punk', 'pixie', 'pop punk', 'post-teen pop'\"],\n",
      "      dtype=object)>, 'artist_genres_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b\"'contemporary country', 'country', 'country road', 'modern country rock'\",\n",
      "       b'NONE'], dtype=object)>, 'artist_name_can': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'Kenny Chesney', b'Metro Station'], dtype=object)>, 'artist_name_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'Chris Young', b'Outasight'], dtype=object)>, 'artist_pop_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([75., 58.], dtype=float32)>, 'artist_pop_seed_track': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([72., 44.], dtype=float32)>, 'artist_uri_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:artist:3grHWM9bx2E9vwJCdlRv9O',\n",
      "       b'spotify:artist:7vXwfZyDp3spzIVNXDaTPN'], dtype=object)>, 'artist_uri_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:artist:4BYxqVkZyFjtik7crYLg5Q',\n",
      "       b'spotify:artist:1zuG3w1Zgeou53fb3Vu3bO'], dtype=object)>, 'collaborative': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'false', b'false'], dtype=object)>, 'description_pl': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'', b''], dtype=object)>, 'duration_ms_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([182960., 179946.], dtype=float32)>, 'duration_seed_track': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([177360., 217733.], dtype=float32)>, 'n_songs_pl': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([124.,  62.], dtype=float32)>, 'name': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'Musiccc', b'Windows Down'], dtype=object)>, 'num_albums_pl': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([83., 58.], dtype=float32)>, 'num_artists_pl': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([54., 52.], dtype=float32)>, 'track_name_can': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'American Kids', b'Shake It'], dtype=object)>, 'track_name_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=array([b\"Love Don't Let Me Down\", b'Now or Never'], dtype=object)>, 'track_pop_can': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([73., 56.], dtype=float32)>, 'track_pop_seed_track': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([31., 37.], dtype=float32)>, 'track_uri_can': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:track:1dgWTMoHwTUnQhOQ8SR5fV',\n",
      "       b'spotify:track:46SFeuNs1K5D76D9vCwFuR'], dtype=object)>, 'track_uri_seed_track': <tf.Tensor: shape=(2,), dtype=string, numpy=\n",
      "array([b'spotify:track:1U8tHnfcubePD1h6i7o4bM',\n",
      "       b'spotify:track:6JHrzpRYiDx53iTgTbI76X'], dtype=object)>, 'album_name_pl': <tf.Tensor: shape=(2, 0), dtype=string, numpy=array([], shape=(2, 0), dtype=object)>, 'artist_genres_pl': <tf.Tensor: shape=(2, 0), dtype=string, numpy=array([], shape=(2, 0), dtype=object)>, 'artist_name_pl': <tf.Tensor: shape=(2, 0), dtype=string, numpy=array([], shape=(2, 0), dtype=object)>, 'artist_pop_pl': <tf.Tensor: shape=(2, 0), dtype=float32, numpy=array([], shape=(2, 0), dtype=float32)>, 'artists_followers_pl': <tf.Tensor: shape=(2, 0), dtype=float32, numpy=array([], shape=(2, 0), dtype=float32)>, 'duration_ms_songs_pl': <tf.Tensor: shape=(2, 0), dtype=float32, numpy=array([], shape=(2, 0), dtype=float32)>, 'track_name_pl': <tf.Tensor: shape=(2, 0), dtype=string, numpy=array([], shape=(2, 0), dtype=object)>, 'track_pop_pl': <tf.Tensor: shape=(2, 0), dtype=float32, numpy=array([], shape=(2, 0), dtype=float32)>, 'track_uri_pl': <tf.Tensor: shape=(2, 0), dtype=string, numpy=array([], shape=(2, 0), dtype=object)>}\n"
     ]
    }
   ],
   "source": [
    "for x in valid_parsed.batch(2).take(1):\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01aeffb3-0126-439f-8eb2-b659e71207a6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-9.m96",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-9:m96"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}