Content Discovery Handbook

Independent websites publish for people first. Search engines, feed readers, directories, and independent crawlers come later—and they only understand what publishers make visible in public HTML, feeds, and related documents.

Open web standards are agreements about how to mark meaning, announce updates, and express access preferences. They let different systems share information without requiring a single company to host everyone’s writing. Content still belongs to its publisher. Standards do not transfer ownership; they make it possible for others to find and interpret what you chose to publish.

Joining Joshternet still requires only a valid /.well-known/josh declaration. Everything in this handbook is optional. The same techniques help across the broader web: personal sites, small publications, and creative portfolios that want to stay portable.

How content discovery works

Discovery is a chain of separate activities. Publishing metadata does not guarantee that any particular system will index, rank, or feature your work.

1. Find Crawlers and people discover URLs through hyperlinks, feeds, sitemaps, and indexes.
2. Retrieve Clients request publicly available resources while respecting applicable access policies.
3. Understand HTML, metadata, microformats2, and structured data describe what was published.
4. Connect Authors, topics, related works, and links establish meaningful relationships.
5. Present Applications may organize what they learned into directories, feeds, search results, graphs, and collections.

Finding a URL is not the same as understanding a page. Understanding a page is not the same as indexing it. Indexing is not the same as presenting it in a topic neighborhood, What’s New strip, or search hit. Each step can succeed or fail on its own.

Chapters

Foundation

• Semantic HTML

  Structure pages with meaningful elements so people, assistive tech, and crawlers can tell articles from chrome. Start here if you want the cheapest win for discoverability on any stack.

• SEO fundamentals

  Titles, descriptions, canonical URLs, and indexing basics that help search engines and independent crawlers decide what a page is about. Useful even when you never chase rankings.

Discovery and publishing

• Slash pages

  IndieWeb path conventions (/now, /uses, /photos, and kin) that organize a site into understandable collections. Helpful when visitors and crawlers need a map, not only a homepage.

• OPML and well-known slashpages

  OPML 2.0 and /.well-known/slashpages as a machine-readable catalog of those collections. Use when you want directories and hubs to enumerate your slash pages without scraping menus.

Describing content

• Microformats2

  Class-based vocabularies (h-entry, h-card, and more) that mark up entries, authors, and categories in HTML. Ideal when you want structure without a separate metadata document.

• Open Graph

  Preview metadata for titles, images, and types that social apps and many readers reuse. Useful when a shared link should look intentional.

• Schema.org and JSON-LD

  Typed structured data in JSON-LD for creative works, events, and other specific kinds of publication. Reach for it when HTML alone cannot name the thing precisely.

Feeds, sitemaps, and crawlers

• Feeds

  RSS, Atom, JSON Feed, and feed autodiscovery for announcing posts and updates. The practical way for readers and bots to learn what changed without re-crawling everything.

• Sitemaps

  XML sitemaps and sitemap indexes that list the URLs you care about discovering. Complements feeds when your site has many stable pages.

• Robots

  robots.txt and related directives that express crawler access policies, plus notes on AI-agent preferences where they exist. Read this before you assume every bot should fetch every path.

Links, identity, and connections

• Links and relations

  Real hyperlinks and link relations (canonical, alternate, and friends) are how URLs get found in the first place. Covers patterns that keep discovery honest without inventing proprietary signals.

• Identity and authorship

  h-card, rel=me, and related patterns that connect pages to a person or project across the independent web. Worth reading when authorship should travel with the work.

• Connections

  Blogrolls, replies, Webmention, and related IndieWeb relationship patterns. For publishers who want meaningful links between sites, not only inbound SEO juice.

Creative publishing

• Creative media

  Photography, music, video, games, and projects as first-class published work. Covers how to describe creative output so directories can list it alongside writing.

• Events

  h-event and Schema.org Event for gatherings with times and places. Useful when your calendar should be readable by people and machines alike.

• Uses

  /uses pages that describe tools and setup without turning into product spam. A small convention with outsized clarity for curious visitors.

• Seeking

  Optional /seeking pages that list wanted help or collaborators; Joshternet can surface explicit requests without becoming a contact directory.

Understanding and troubleshooting

• Explore

  How to make writing and creative work easier to discover—subjects, feeds, stable URLs, and structured metadata—without treating discovery as a guarantee.

• Validation

  Checklists, validators, and debugging steps for feeds, markup, and declarations. Start here when something looks right locally but fails in the wild.

• Reference

  Specifications index and verified hub crawl matrix for the Content Discovery Handbook. Keep it bookmarked for authoritative links and honest coverage notes.

How the specifications work together

Imagine Harbor Notes, a fictional independent site run by one person. They publish a long article about rebuilding a workbench and a photography project under /photos/.

• Semantic HTML marks the article as an article, the photo set as a collection of work, and dates as dates—so browsers and extractors are not guessing from CSS classes alone.
• Links and sitemaps make those URLs discoverable from the homepage, related posts, and an explicit URL list.
• Feeds announce the new article (and later updates) to readers and bots that prefer “what changed” over a full recrawl.
• Microformats2 describes the entry, author, and categories in the HTML itself.
• Open Graph supplies a title, description, and image for previews when the article is shared.
• Schema.org and JSON-LD can name the photography project as a creative work with a clearer type than a bare blog post.
• Slash pages keep /photos/ (and similar roots) as understandable collections instead of a pile of undated files.
• OPML and /.well-known/slashpages can advertise those collections in a machine-readable index.
• Robots express which paths crawlers may fetch; additional AI-agent or content-usage directives may express other preferences where they are supported.

None of these steps replace the others. HTML remains the foundation. Feeds and sitemaps improve discovery. Structured markup improves interpretation. Access policies set boundaries. Presentation layers—Joshternet’s Topics and Showcase included—still choose what to show.

Recommended implementation path

Adopt standards in an order that matches ordinary site work. You do not need every chapter before your content can be found.

1. Start with the basics. Publish semantic HTML, descriptive titles, real links, and meaningful URLs (SEO fundamentals).
2. Make content discoverable. Add feeds, sitemaps, and appropriate crawler policies.
3. Describe your work. Add microformats2, Open Graph, or Schema.org where the content warrants it—and identity when authorship should travel with the page.
4. Organize your site. Introduce slash pages and optionally publish an OPML catalog.
5. Connect with others. Publish meaningful outbound links, blogrolls, and supported IndieWeb relationships. Creative media, events, and uses pages fit here when they are part of what you publish.
6. Validate and discover. Confirm resources and metadata with the validation checklists, then use the reference when you need authoritative links or honest crawl coverage notes. Explore covers how to make published work easier to find—not a guarantee of prominence.

Also

• Implement — the declaration
• Connections crawl — maintainer pipeline
• JoshBot
• Join