hckrnws

Hello!

I wanted to share my recent project: Tadpole. It is a custom DSL built on top of KDL specifically for web scraping and browser automation. I wanted there to be a standardized way of writing scrapers and reusing existing scraper logic. This was my solution.

Why?

    Abstraction: Simulating realistic human behavior (bezier curves, easing) through high-level composed actions.
    Zero Config: Import and share scraper modules directly via Git, bypass NPM/Registry overhead.
    Reusability: Actions and evaluators can be composed through slots to create more complex workflows.

This is a fully running example, @tadpole/cli is published on npm:

tadpole run redfin.kdl --input '{"text": "Seattle, WA"}' --auto --output output.json

  import "modules/redfin/mod.kdl" repo="github.com/tadpolehq/community"

  main {
    new_page {
      redfin.search text="=text"
      wait_until
      redfin.extract_from_card extract_to="addresses" {
        address {
          redfin.extract_address_from_card
        }
      }
    }
  }

    Control Flow: Add maybe (effectively try/catch) and loop (while {}, do {})
    DOMPick: Used to select elements by index
    DOMFilter: Used to filter elements using evaluators
    More Evaluators: Type casting, regex, exists
    Root Slots: Support for top level dynamic placeholders
    Error Reporting: More robust error reporting
    Logging: More consistent logging from actions and add log action to global registry

    Piping: Allowing different files to chain input/output.
    Outputs: Complex output sinks to databases, s3, kafka, etc.
    DAGs: Use directed acylic graphs to create complex crawling scenarios and parallel compute.

I've also created a community repository for sharing scraper logic: https://github.com/tadpolehq/community

Feedback would be greatly appreciated!

I like the idea of a DSL for scraping but my scrapers do more than extract text. I also download files (+monitor download progress) and intercept images (+ check for partial or failed to load images). So it seems my use case isn't really covered with this.

Thanks for the idea actually! It's difficult to cover every use case in the 0.1.0 release. I'll take this into account. Downloading Files/Images could likely be abstracted into just an HTTP source and the data sources could be merged in some way.

I had to look up what KDL is and what `Functional Source License, Version 1.1, ALv2 Future License` is.

So KDL is like another JSON or Yaml. FSL-1.1-ALv2 is an, almost but not really, open source license that after a 2 years becomes available under a real open source license. It's to prevent free loading from companies or something. Sounds fine to me actually.

Effectively, it's not meant to restrict people from using it, even in a commercial setting, just to protect my personal interests in what I want to do with it in a commercial setting.

KDL is more than just JSON or YAML. It's node based. It's output in libraries is effectively an AST and its use cases are open ended.

For me the acronym clash was "DSL". Presumably a language for sucking information off web pages.

Domain Specific Language - It's a fairly common acronym I feel like. SQL is a DSL, so is CSS, Rust macros lets you create DSLs, for example. The opposite is General Purpose, like Python, JavaScript

I really enjoy this as a concept! I run a service for away-ordering food from vendors at my University, and I need to do a lot of web-scraping to facilitate this. The code example on the website looks a lot cleaner than any of my BS4 code...! Even with the minimal primitives implemented, this seems useful.

Most of my webscraping code is in Python, do you have plans to implement bindings to other languages?

I'm glad you like the project!

You know that's an interesting idea, NodeJS does support WASM. Compiling Python to WASM is possible. I could add custom actions for importing and executing external code. We'll see, it's given me something to think about!

You know a better dsl? jquery.

Not really. Operating with Chrome Devtools Protocol over WS is completely different. You cannot move your mouse, you cannot open tabs, you cannot control the browser in the same way. It's not really comparable, sorry.

Here is a reference for you to read up on: https://chromedevtools.github.io/devtools-protocol/

what is the specific domain of jquery ?>

I don't think he understands what CDP is vs Client Side JavaScript.