I am very confused about the Algolia Crawler Script. Any help is appreciated.

[themagicalmammal]

This is my crawler code, I have experimented with everything. I have used multiple vitepress configs the new infrastructure one the old one, but I am still confused on why I can't index headers but with vitepress website there crawler is working. I have tried almost everything at this point and I get more confused when I read the documentation.

Any help is appreciated. The website I am trying to crawl is https://howtolinux.vercel.app.

new Crawler({
  rateLimit: 12,
  maxDepth: 16,
  startUrls: ["https://howtolinux.vercel.app/"],
  renderJavaScript: true,
  sitemaps: [],
  ignoreCanonicalTo: false,
  discoveryPatterns: ["https://howtolinux.vercel.app/**"],
  schedule: "at 2:42 PM on Monday",
  actions: [
    {
      indexName: "howtolinux",
      pathsToMatch: ["https://howtolinux.vercel.app/**"],
      recordExtractor: ({ helpers }) => {
        return helpers.docsearch({
          recordProps: {
            lvl1: [
              "header h1",
              "article h1",
              "main h1",
              "h1",
              "head > title",
              ".content h1",
            ],
            content: [
              "article p, article li",
              "main p, main li",
              "p, li",
              ".content p, .content li",
            ],
            lvl0: {
              selectors: "",
              defaultValue: "Documentation",
            },
            lvl2: ["article h2", "main h2", "h2", ".content h2"],
            lvl3: ["article h3", "main h3", "h3", ".content h3"],
            lvl4: ["article h4", "main h4", "h4", ".content h4"],
            lvl5: ["article h5", "main h5", "h5", ".content h5"],
            lvl6: ["article h6", "main h6", "h6", ".content h6"],
          },
          indexHeadings: true,
        });
      },
    },
  ],
  initialIndexSettings: {
    howtolinux: {
      attributesForFaceting: ["type", "lang", "tags"],
      attributesToRetrieve: ["hierarchy", "content", "anchor", "url"],
      attributesToHighlight: ["hierarchy", "hierarchy_camel", "content"],
      attributesToSnippet: ["content:10"],
      camelCaseAttributes: ["hierarchy", "hierarchy_radio", "content"],
      searchableAttributes: [
        "unordered(hierarchy_radio_camel.lvl0)",
        "unordered(hierarchy_radio.lvl0)",
        "unordered(hierarchy_radio_camel.lvl1)",
        "unordered(hierarchy_radio.lvl1)",
        "unordered(hierarchy_radio_camel.lvl2)",
        "unordered(hierarchy_radio.lvl2)",
        "unordered(hierarchy_radio_camel.lvl3)",
        "unordered(hierarchy_radio.lvl3)",
        "unordered(hierarchy_radio_camel.lvl4)",
        "unordered(hierarchy_radio.lvl4)",
        "unordered(hierarchy_radio_camel.lvl5)",
        "unordered(hierarchy_radio.lvl5)",
        "unordered(hierarchy_radio_camel.lvl6)",
        "unordered(hierarchy_radio.lvl6)",
        "unordered(hierarchy_camel.lvl0)",
        "unordered(hierarchy.lvl0)",
        "unordered(hierarchy_camel.lvl1)",
        "unordered(hierarchy.lvl1)",
        "unordered(hierarchy_camel.lvl2)",
        "unordered(hierarchy.lvl2)",
        "unordered(hierarchy_camel.lvl3)",
        "unordered(hierarchy.lvl3)",
        "unordered(hierarchy_camel.lvl4)",
        "unordered(hierarchy.lvl4)",
        "unordered(hierarchy_camel.lvl5)",
        "unordered(hierarchy.lvl5)",
        "unordered(hierarchy_camel.lvl6)",
        "unordered(hierarchy.lvl6)",
        "content",
      ],
      distinct: true,
      attributeForDistinct: "url",
      customRanking: [
        "desc(weight.pageRank)",
        "desc(weight.level)",
        "asc(weight.position)",
      ],
      ranking: [
        "words",
        "filters",
        "typo",
        "attribute",
        "proximity",
        "exact",
        "custom",
      ],
      highlightPreTag: '<span class="algolia-docsearch-suggestion--highlight">',
      highlightPostTag: "</span>",
      minWordSizefor1Typo: 3,
      minWordSizefor2Typos: 7,
      allowTyposOnNumericTokens: true,
      minProximity: 1,
      ignorePlurals: false,
      advancedSyntax: true,
      attributeCriteriaComputedByMinProximity: true,
      removeWordsIfNoResults: "allOptional",
    },
  },
});

[shortcuts]

Hey! I think what you mean here is that it will always return Documentation as the main section in the search right?

In order to fix that, you need to move your lvl* selectors in the recordProps to the one before, e.g. lvl1 becomes lvl0 and so on. The content field remain the same.

Hope this solves your issue :)

[randombeeper]

I'm going to close this discussion as we are trying to keep interactions to our Discord at https://alg.li/discord and we'll be happy to help you there! This discussion area is not well monitored.