this post was submitted on 16 Jun 2023
44 points (85.5% liked)
Asklemmy
43526 readers
2582 users here now
A loosely moderated place to ask open-ended questions
Search asklemmy ๐
If your post meets the following criteria, it's welcome here!
- Open-ended question
- Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
- Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
- Not ad nauseam inducing: please make sure it is a question that would be new to most members
- An actual topic of discussion
Looking for support?
Looking for a community?
- Lemmyverse: community search
- sub.rehab: maps old subreddits to fediverse options, marks official as such
- [email protected]: a community for finding communities
~Icon~ ~by~ ~@Double_[email protected]~
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
If you wanna write code to do this ... I'd say skip the bot, write a gateway instead.
Back in the early days of email, there were lots of different email systems, not just the SMTP Internet email we use today. There was UUCP email with "bang paths", where your email address specified a list of servers that a message could be passed through to get to you. There were other networks like FidoNet and WWIVnet, that could send email to Internet email addresses through special "gateway" servers.
A gateway receives messages using one protocol or service, and retransmits or makes them available on another protocol or service.
For a little while in 1992, I had access to read Usenet posts only through a gateway that exported Usenet posts onto the Gopher system.
A gateway between Reddit and Lemmy would appear to Reddit as a web browser, scraping posts and comments; while appearing to Lemmy as a Lemmy instance that users could subscribe to, making each subreddit it scrapes available as a Lemmy community.
So a Lemmy user could subscribe to, say, [email protected] and see a fresh view of AskReddit. The server at reddittolemmy.com would not be a standard Lemmy server with users, but rather a custom gateway server that fetches data from Reddit and makes it available in the form of a Lemmy community.
(If Reddit were not being an asshole, a gateway could be an API client. But Reddit is being an asshole, so a gateway should probably be written as a scraper that accesses Reddit as if it were a normal user using a desktop Web browser.)
This is a great idea.
I don't particularly think the whole of Reddit needs to be scraped though. I could be happy with only scraping posts that pass a certain thresh hold of votes against the subreddits subscriber count and maybe getting those crossposted to the Lemmy equivalent communities that want to opt in to such a service. This would be especially useful for World News and the more niche subreddits that don't yet have a big enough userbase here
The hard part isn't describing which posts or comments need to be gatewayed.
The hard part is being able to deliver posts and comments across the gateway at all.