This repository has been archived by the owner on Jun 30, 2024. It is now read-only.

3n3a-archive / robotstxt-parser Public archive

Notifications You must be signed in to change notification settings
Fork 0
Star 0

A Robots.Txt Parser available as Golang Module

0 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
_examples/http-request-parse		_examples/http-request-parse
.gitignore		.gitignore
.tool-versions		.tool-versions
LICENSE		LICENSE
README.md		README.md
RobotsTxtParser.go		RobotsTxtParser.go
RobotsTxtParser_test.go		RobotsTxtParser_test.go
go.mod		go.mod
go.sum		go.sum

Repository files navigation

robotstxt-parser

A Robots.Txt Parser available as Golang Module

Features

Different parsing of Robots.txt (from Ua to UA and Sitemap separate)

About

A Robots.Txt Parser available as Golang Module

Custom properties

Report repository

Releases 1

Packages

No packages published

Languages

Go 100.0%