I have written a search engine that works just like google but 100x safer and with no trackers, right now my database has over 3,238,313 rows of data yet is under 1/2 of a GB. It is under construction at
https://sturtz.cf it is written in PHP and MySQL. I wrote it to get my family off of Google and never thought it would go any where ever. But I working at it and here I am, growing by over 1 million links a day. You can find the Source at
https://git.sturtz.cf Thank you for reading this. :)
The Crawler works but finding the URL with: (
https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...)
if (mbsubstr($element->href, 0, 2) == "//") {
$element->href = $urlparts['scheme'].":".$element->href;
}
if (mbsubstr($element->href, 0, 1) == "/") {
$element->href = $base_url.$element->href;
}
if (mbsubstr($element->href, 0, 2) == "./") {
$element->href = substr($element->href, 2);
}
if (mbsubstr($element->href, 0, 5) != "http:"
&& mbsubstr($element->href, 0, 6) != "https:"
) {
$element->href = $base_url.$url_path.$element->href;
}
which looks for the href tag and will send the spider there to crawl it
then it puts it into the database by some INSERT SQL commands: (
https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...)
if ($md5sum == '') {
$stmt = $db->prepare(
"INSERT INTO `".$mysql_table_prefix
."links` (site_id, url, title, description, fulltxt, "
."indexdate, size, md5sum, level) VALUES ( ? , ? , ? ,"
." ? , ? , CURDATE(), ? , ? , ? )"
);
You can find a working version at https://sturtz.cf
if ($md5sum == '') { $stmt = $db->prepare( "INSERT INTO `".$mysql_table_prefix ."links` (site_id, url, title, description, fulltxt, " ."indexdate, size, md5sum, level) VALUES ( ? , ? , ? ," ." ? , ? , CURDATE(), ? , ? , ? )" ); You can find a working version at https://sturtz.cf