About a decade ago, I tried to index the web via the Dewey Decimal System. I had a site laid out similar to Google, where you could browse sites continuously starting from a given call, but the DDS is proprietary, and those people hate anyone who uses their IP without a license. You can Google everyone they’ve shutdown – places that weren’t even libraries – for using anything similar to the DDS. I reached out to the group that manages the DDS, and was taken offline before my project even started.
With all the corporate BS lately, and people looking for alternative options, I thought I’d take a search engine old school, and we’d index like Usenet. Except with an XXX.XXX.XXX format.
My original project used sharded Redis, with append logs to disk. I chose it for its in-memory speed and key-value store. Did some calculations, and I’d have to have millions of records just to consume my entire system’s memory.
I had a lot of plans for this before getting shut down.
Now that I’m older, I’m curious if I should be using MongoDB.
What are the benefits and drawbacks of each? Which would you use? And why?
Depends on the data you want to store and what you want to do with it. I haven’t used either MongoDB or Redis in particular, but MongoDB (or e.g. CouchDB) is a document-oriented database, whereas Redis/Valkey is a key-value store.
Superficially, these are similar kinds of databases, in that you have a key and store a value. But document-oriented typically enforces the format somewhat, like e.g. some JSON format, which is then used for doing indexing, aggregation, search or whatever in the database.
Meanwhile, a key-value store does not care what data is actually stored inside. It can literally be just some bytes. You tell it the key and it tells you the data you previously stored, and that’s all you do with it.
I wouldn’t be surprised, though, if Redis doesn’t fit that definition 100% and actually has some optional document-oriented features as well.
From your very short description of what you want to do, my intuition would be more towards document-oriented, because the indexing is presumably essential for a search engine.
But as someone else already said, MongoDB is proprietary now, so probably want to look for an alternative. Selecting the database is worth spending some time on, since the right database might already bring along the search capabilities you want.
I also want to throw in a third type of database that you should probably look into: https://en.wikipedia.org/wiki/Graph_database
Graph databases are good at storing relationships between entries, which you will have a lot of, given each webpage links to other webpages. I believe, the big-boy search engines do use graph databases.