Skip to content

Commit

Permalink
Add log messages to scrape bills introduced to better track scraping …
Browse files Browse the repository at this point in the history
…progress
  • Loading branch information
limdingwen committed Jul 25, 2024
1 parent 7bf77b2 commit cd65c5e
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions supabase/functions/scrape-bills-introduced/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Deno.serve(async (req) => {
return buildResponse({ message: "Unauthorised." }, 401);
}

console.log("Getting bills introduced HTML from URL...");
const billsIntroducedUrl =
"https://www.parliament.gov.sg/parliamentary-business/bills-introduced";
const billsIntroducedPaginationSize = "40";
Expand All @@ -33,6 +34,7 @@ Deno.serve(async (req) => {
body: new URLSearchParams({ PageSize: billsIntroducedPaginationSize }),
});

console.log("Parsing the downloaded HTML...");
const billsIntroducedHtml = await billsIntroducedResponse.text();
const billsIntroducedDoc = new DOMParser().parseFromString(
billsIntroducedHtml,
Expand All @@ -42,6 +44,7 @@ Deno.serve(async (req) => {
let addCount = 0;
let updateCount = 0;

console.log("Scraping and uploading relevant data...");
const billsIntroduced = billsIntroducedDoc.querySelectorAll(
".indv-bill",
) as Iterable<Element>;
Expand Down

0 comments on commit cd65c5e

Please sign in to comment.