Skip to content
Fito

Fetch documentation from a URL

When a technology has no code for you to point at — a practice, say, or a tool whose useful knowledge lives in its documentation — you feed the knowledge pipeline a list of URLs instead. The fetch turns each page or video into a cleaned markdown file that the identify commands and the technology draft can read.

This page covers writing the URL list, running the fetch, and reading what it leaves behind. It is the fallback path. When you have a project on disk, a source tree is the better input — see Ingest a source tree.

Before you begin

Section titled “Before you begin”
  • A pi session in a working directory of your own, with Fito installed there. See Set up a working directory.
  • An LLM provider configured in pi. The fetch is carried out by your session, which writes and runs a small Python script to do it.
  • python3 able to create a virtual environment (python3 -m venv). The fetch installs its Python packages there rather than into your system.
  • Network access to every URL on the list.

You do not need to be logged in to Fito. The fetch never talks to the Fito server.

Write the URL list

Section titled “Write the URL list”

Put one URL per line in a text file. Blank lines are skipped, a line starting with # is a comment, and a URL that appears twice is fetched once.

# Picocli
https://picocli.info/quick-guide.html
https://github.com/remkop/picocli#readme

Two kinds of URL are told apart:

  • A YouTube video — a youtube.com/watch, youtu.be/ or youtube.com/shorts/ link. Its transcript is fetched.
  • Everything else — fetched as a web page.

Fetch the list

Section titled “Fetch the list”

  1. Save the list in your working directory. The default name is urls.txt.

  2. In your pi session, run:

    /fito-fetch-urls-content urls.txt

    Replace urls.txt with your list’s path. With no argument, the command looks for urls.txt in the working directory. The command takes no flags: the list’s path is its only argument.

  3. Read the summary it ends on: how many URLs succeeded, how many failed, and the error for each failure.

You can also fetch as the first part of another command. Both the identify commands and /fito-create-or-update-technology take --fetch, with the list named by --urls:

/fito-identify-domain-concepts Picocli --technology picocli --technology-version 4 --fetch --urls urls.txt

--urls defaults to urls.txt. Pass the file with --urls, not as a value of --fetch. The identify command also needs the technology provisioned first — here, picocli@4. See Provision a technology.

Confirm it worked: one markdown file per URL that resolved sits in ./output/1-ingest/extracted_links_content/. Any URL that produced no file is listed in _errors.md in the same folder, with the reason.

What each file holds

Section titled “What each file holds”

Each file is named after the page’s title, lower-cased and dash-separated, and opens with a frontmatter block:

Field What it holds
source_url The URL it came from
title The page or video title
type webpage or youtube
fetched_at When it was fetched, in ISO 8601
author The author, when the page names one
summary A summary of the content in your session’s own words, 50 words at most

The body is the page’s main text as markdown: headings, inline links, lists, blockquotes and code blocks. Navigation, footers, sidebars, ads, cookie banners, social buttons and sign-up blocks are removed. A transcript is broken into paragraphs, with the duplicate lines of auto-generated captions removed.

A transcript is kept in whatever language it has. It is not translated, and no note is added about its language or where it came from.

How the fetch behaves

Section titled “How the fetch behaves”
  • Web pages go through a content-extraction library. A page that comes back with 200 characters or fewer of real content — or with far less than its title promises — is treated as rendered by JavaScript. The fetch retries it through a renderer, and parses the plain HTML only as a last resort.
  • Videos use the published transcript, preferring manual captions to automatic ones. When there is no transcript at all, the fetch falls back to the auto-generated subtitles.
  • Each web request times out after 15 seconds.

It leaves no trace. The virtual environment, the generated script and any subtitle file live in a scratch directory outside your working directory, and the run removes that directory at the end — after a failure as well as after a success. The only thing the run adds to your working directory is the documents it writes. Anything already there, including other stages under output/ and your URL list, is left alone.

Read the documents in

Section titled “Read the documents in”

The identify commands read the fetched folder by default: their --docs flag defaults to ./output/1-ingest/extracted_links_content/. So after a fetch, run an identify command with neither --source nor --fetch:

/fito-identify-domain-concepts Picocli --technology picocli --technology-version 4

/fito-create-or-update-technology reads the same folder by default, so one fetch can ground both the technology’s descriptions and its components.

When a URL fails

Section titled “When a URL fails”

A URL that produces no file is written to _errors.md with the reason, and the fetch moves on to the next one. One bad URL never stops the run.

A transcript in another language is not a failure, and neither is a page whose extraction needed the fallback. Only a URL that ended with no file is logged.

When a URL fails, open it in a browser before you fetch again.

What’s next

Section titled “What’s next”