Reddit Data API Update: Changes to Pushshift Access

Does this destroy tools like removeddit? Because I use that website constantly.

It's cute that you can't show us "user-deleted content" for "lawyercat reasons," but when I have someone with 1,600+ karma post in my subreddit, and every single other post or comment that person has made has been deleted, I use those tools to see if they have posted in multiple R4Rs across the country on the same day. This is the most reliable way for me verify if I'm dealing with a karma farmer.

I also use these tools to verify the authenticity of an account. I have caught minors who tried to post in my NSFW sub because I recovered posts that had their *real* age. Spammers and scammers rarely stand up to a cursory search using a Pushshift app. In many cases, these accounts are ridiculously fluid about things like age and gender. But Reddit offers no protections to moderators to prevent these accounts from changing any detail they'd like, as long as they delete their postings.

You are removing a vital tool with absolutely no replacement ready, and that is absolutely unfair to those of us who are volunteering to moderate the content on your platform. Moderation tools, at this point, should be moving forward, but Reddit is about to throw the moderators **YEARS** backwards, while the scammers, spammers, and bots continue to find new and exciting ways to spam our subreddits—which the moderators take the heat for if we fail to adequately protect the sub.

I already spend several hours a day moderating my subreddits (and I think I do a damned good job). If Reddit continues to impede my abilities as a moderator, I see no reason to keep the subreddits running. I'll take them all private, and let them die.

Mod of r/pushshift here, I've crossposted this to the subreddit for further discussion: https://www.reddit.com/r/pushshift/comments/134uvzz/reddit_data_api_update_changes_to_pushshift/

[deleted]

The maintainer of Pushshift has been dealing with some stuff recently and their availability has been affected.

None of these "improvements" address the reasons why we are all using Pushshift for both subreddit bots and general moderation bots such as BotDefense, repost detection bots, and others.

Reddit is doomed to be overrun by even more reposted content, both submissions and comments, from malicious bots because they do not care about the Reddit terms of use and they will definitely not have a problem sourcing old content without Pushshift.

I think we will need to reconsider whether it is worth continuing BotDefense now that it has been crippled by this awful decision.

for lawyercat reasons

Smile at our jokes while we screw over our volunteer moderator labor force?

I am not laughing.

There are so many uses for pushshift and ban flow/removal reasons are at the bottom of that list

Hey u/lift_ticket83 -- I apologize for the communications gap and not being responsive when trying to contact us. There was some internal issues and confusion on who was supposed to handle comms while I deal with family issues. I'm happy to jump on a call with you to discuss where we are deficient and how we can meet your API terms.

As you know, Pushshift is used extensively in the academic community and I have always made a good faith effort to honor user requests when a user makes a request. In fact, we now do this daily.

Could you give me some contact information so we can set up a meeting with our team and your team to discuss the best path forward?

Thanks again and I apologize for the the issue with comms.

In summary, it seems you all don't care about the fact that mods and others use Pushshift for good to combat spam and malicious bots, and don't want to provide similar functionality for cost reasons.

I can't wait until this backfires. Enjoy your future IPO though.

This just made modding 100x harder. Thanks.

Can you first natively implement the tools we depend on for moderating before nuking something almost all serious mods rely on?

This is going to seriously affect our workflow.

Providing permalinks to user- and admin-deleted content in User Mod Log for any given user in your community. Please note that we cannot show you the user-deleted content for lawyercat reasons.

As has been previously discussed, this is still insufficient. Continued access to the user-deleted content is incredibly important because otherwise users can just delete offending content to try to hide their tracks. Yes, we can add notes (natively or via toolbox) but that requires additional effort for moderators.

Thank you for killing off a useful tool many of us use daily.

our team has been working diligently to understand your usage of Pushshift functionality to provide you with alternatives within our native tools in order to supplement your moderator workflow. Some improvements we are considering include:

One critical pusshift.io feature that I will miss is a comprehensive, historical list of submissions to the subreddits I moderate, and it sounds like there is no replacement in the works.

Currently, Reddit only provides an extremely limited submission history for a subreddit, of a hundred pages or so. I've previously used year-long periods to train Automoderator filters, using my own (compliant) API access to process comments into a training/test corpus when given a list of submissions.

I don't care much about the permanence of the historical record since a vanishingly small fraction of submissions are edited away or deleted; I do care about finding out what threads were posted six months ago.

to be honest, it seems like the changes to the API terms were seemingly primarily targeted at Pushshift from the beginning

Comment bots celebrating right now.

I'm surprised this post seems to tie development of mod tools directly to third party tooling.

This adds pressure on developers, like of /r/toolbox etc., to go dark so tools get developed natively in a timely manner.

Clearly admin aren't prioritizing mod tools sufficiently if going dark impacts mod tooling in any way whatsoever.

Needing third party tools to do basic community management is a pretty critical vulnerability for running the site no?

Why did you do this now instead of notifying everyone you would and letting people try to migrate tools? I have a meeting with you guys literally this week to talk about all the ways I use pushshift.

Not to mention I specifically asked if you had talked with the pushshift owner and I was told you had, so I anticipated more time.

Put up a historical/research API endpoint with hyperspecific parameters like pushshift and gate access to institutions/big mods/phone interviews/NDAs/etc (hell, I'll even PAY for access!), otherwise you've just kneecapped mods only way to combat platform manipulation of every kind.

Edit: Research institutions, governments, digital forensics outfits all relied on Pushshift for historical data and the ability to craft hyperspecific requests -- have you spoken to any of them? How can they study or help your platform at all anymore?? How can we? Since day one, people have had to kludge extensions and sites built around your own because of its complete lack of functionality. Killing Pushshift and not having a replacement is insanely on-brand for ya'll.

Why are you turning this off with ZERO warning to us? Why do you not have replacement tooling for us to migrate to? This is critical functionality that you are pulling out from under us, functionality that we rely on to do our volunteer jobs, and the best we get is "howdy mods" and "lawyercat" with a side of patronizing tone? This is an incredible slap in the face, especially to those of us who have been working with you all in good faith on detailing what Pushshift means to us. I'm honestly gobsmacked.

So basically you'll be destroying our moderating capabilities just because you can't manage to get in touch with one guy.

Considering how we routinely weed out spam and whatnot and deal with all sorts of rule-breaking behavior, axing the access to Pushshift ensures that all of those will be absolutely rampant.

You also clearly didn't "hoped to avoid" disruptions with a last minute notice like that. If anything a behavior like this on your behalf is the polar opposite of that.

Given the IPO rumors, it seems that the enshittification of Reddit is well under way...

Knew this was coming and it's still extremely painful, this is going to make moderating so much harder.

I just want to say 'fuck this shit' with my whole chest.

I moderate a question and answers type subreddit and we use several tools that allow us to deep dive into past asked questions to find similar already asked questions, to create FAQ pages, statistics, in general go through old posts.

Reddit API as far as I can tell no longer supports search by before and after dates. Pushshift does and was indispensable tool for it. Reddit API also has some limits when going into past but as I haven't used it much I don't remember which ones exactly.

Do you plan to introduce (back) the ability to search by dates, and allow us easy search of old posts and comments by different criteria (subreddit , author, date, etc.)

If you believe such use cases are covered by your API let me know how

The original post content no longer exists here. The author used Redact to remove it, exercising their right to control their data & privacy.

Chunky fly velvet noodle cheerful imagine

Hello.

Is it possible to get an update on Reddit's plans for how academics and non-profit researchers will be impacted by both this and Reddit's API changes.

Pushshift has been invaluable to the research community. One of the reasons I have been impressed by Reddit is their previous interest in allowing academics and researchers to use Reddit data (directly via the API and pushshift) to conduct research. However, this looks like researchers will be essentially shut out completely overnight, which is really disappointing.

Can someone please give an update as to what we can expect for researchers?

This is the worst decision Reddit has made in literally years. The concerns about privacy are valid but there must be a way to address those without dealing such a massive negative blow to critical mod tools, powerful sitewide search, academic projects, and more. It's a hasty and terrible move that has got to be reversed or at least rethought, especially now that /u/stuck_in_the_matrix is back and responsive (despite dealing with a nightmare set of IRL medical problems). Saying "thanks!!!" to mods and third-party devs is easy, actually accommodating their needs instead of railroading them when it's convenient proves you mean it. Do the right thing here, or there's potential for one of the biggest power user backlashes ever, not a great thing to court right before a splashy public offering. Remember what happened to Digg?

Standard reddit m.o.:

• Remove/break useful feature/workflow
• Promise vague improvements that never actually materialise

For an example, look no further than the big "PRO CSS" banner in the sidebar here. 5 years later...

If you'd just be honest and say you're doing this to make more money and you don't really give a shit about the impact, I'd have more respect for all of you.

As many other people have pointed out in this thread, the use cases you describe are way too narrow when compared to the functionality that PushShift provided. The most glaring one being the ability to overcome the 60 requests/min rate limit. How do you expect moderators of large communities to build tools when they get more than 60 comments/min? AutoModerator is great, but it has so many gaps in it's functionality when compared to the custom tools that moderators build to manage large communities.

I have poured 100s of hours of my own free time into building a bot for /r/CryptoCurrency that flair's users based on their account history. I heavily relied on PushShift to help me gather large amounts of historical data on users in order to derive their flair. After this change I see no way forward for my bot to continue functioning. The feature is novel, but I feel that it adds a lot to the experience in the community. Many communities rely much more heavily on similar types of bots, and it's clear based on your listed use cases that you have consciously disregarded them. Acting like those options do anything to cover the functionality that PushShift provided is disingenuous at best.

This can only be described as a slap in the face to the moderators that run your site for you. Did Reddit forget how much they rely on mods who work for free? Did Reddit forget what happened in 2015 when they pushed moderators too far? I really want to be optimistic and believe that Reddit will develop a sufficient alternative, but anyone who has been paying attention long enough can't be anything but pessimistic by this point. Nothing Reddit has said has assuaged any concerns us mods share.

If there is no replacement for PushShift then I'm done. My work will go into the trash. I'm sure many other moderators feel the same way right now. Reddit relies on our labor to function, and I will not be dedicating any more of my free time to a site that has so little respect for the work that we do.

[removed]

Not everyone needs to respond to the administrators all of the time. Push shift is one of the most important tools for multiple subreddits of mine. This is how we keep users from stealing and scamming, and it will break several other subreddits.

Would it hurt you to just not shut it off? And allow it to continue regardless of whether they speak with you?

You know what else really sucks about this?

I used pushshift a few months ago to find my first comment / post on reddit, made 11 years ago. I then used it to find a couple of posts that I had made many years ago but couldn't find any other way, but I wanted to show someone.

It was nice to be able to go back through my account for nostalgia reasons. That simply will not be possible anymore.

Now there is simply no way to go back through your own account to find year's old posts.

There is a clear difference between user-reason: 'I removed my comment because I felt embarrassed by it.' and user-reason: 'I removed my comment because I'm trying to evade the Mods, knowing that this comment is inappropriate for this subreddit.' We need helpful tools for the latter -- and the PushShift provided that. The Admins here, however, have significantly failed in providing such tools, and removing the one useful tool we did have -- abruptly, and with no warning -- constitutes yet another failure of the Admins to support Moderators.

This is disastrous in many ways.

I will never again be able to revisit a mod action taken on an item a user deleted.

After 90 days I can no longer revisit a mod action on an item AEO removed.

It is the reddit users who are the victims of this change and it will absolutely cause people to be stuck with bans they can not possibly appeal.

Guys. This is really, really bad.

Some improvements we are considering include:

• No longer make it impossible for mods to see the content of an AEO removal after 90 days.
• Improve native search tools & streamline the results.

I'm happy you're pushing out enhancements for us as quickly as you're shutting down tools.

Enhancing “removal reasons” by untying them from user notifications.

Please make this “and”, not “or” if you decide to build this.

Updating the ban flow to allow mods to provide additional “ban context” that may include the specific content that merited the user’s ban.

Yes, please.

Talk about tonal whiplash.

The post about reducing 3rd party access to the API tries to keep it positive by discussing how this "will (and won’t!) impact moderators" and highlighting how instrumental 3rd party tools are to moderating on Reddit:

Before you ask, let’s discuss how this update will (and won’t!) impact moderators. We know that our developer community is essential to the success of the Reddit platform and, in particular, mods. In fact, a HUGE thank you to all the developers and mod bot creators for all the work you’ve done over the years.

Not even 2 weeks later, Pushshift is being shut down without any public forewarning. A tool that a bunch of people called out as being important to how they help moderate reddit.

^{And at least one person predicted that this exact thing would happen}.

This is not good in a very big way. Might be time to consider getting out of the moderator game.

[deleted]

for lawyercat reasons.

You stopped being a relatable start-up some years ago, you're a corp now. Your actions are there, work on your communications as you seem confused.

For newer mods that don't know what Pushshift is:

rude instinctive cough sophisticated pie air deliver different ludicrous distinct

This post was mass deleted and anonymized with Redact

How do you guys seem to miss the mark every single time. 🙃

Among all the unfavorable actions taken by Reddit in the past, this particular one will have the most significant and far-reaching consequences in all of my communities. As a moderator, I am now grappling with the difficult decision of whether it is still worthwhile to continue curating our communities, as the loss of this critical tool will severely impact our ability to distinguish genuine users from fraudsters and dishonest individuals who only seek to deceive and exploit others.

Stripping us of the one invaluable tool that we had at our disposal, has ultimately compelled us to increase our subreddit eligibility standards. Thanks to this decision, it has now become much more challenging for deserving individuals to find the help they need.

This is an awful change. I moderate subreddits that have been around for nearly a decade. There is no way of accessing and indexing old content in the absence of PushShift, reddit's native filters and search are awful in this regard.

I don't think you understand how much of this site's operation relies on PushShift, either directly or indirectly. You risk killing off the site with this change, and nobody will want to fill the void knowing capricious management might kill their efforts in a few months if they grow too large.

Watching Reddit die here... in real time

You know who loves this? SPAMMERS.

This accomplishes nothing but further deteriorate Reddit.

Take our tools to fight spam away add eventually the subs aren't going to exist. With all the alternatives out there, this is just a really poor long-term businesses decision.

[deleted]

hopefully the admins reconsider this mistake.

harming pushshift is in turn harming reddit by making the site substantially worse for both moderators and regular members alike.

pushshift is by far the best way to search comments, and more importantly it brings a vital piece of transparency to reddit.

while i will remove rule breaking posts in my sub, i explicitly tell my members to feel free to look on reveddit and unddit (both of which rely on pushshift) so they can transparently see what is being removed and can judge for themselves if the removal is fair. i want my users to be able to see what i'm doing rather than having everything obfuscated.

if money is an issue for pushshift, you should be gifting them a premium api subscription akin to how elon musk gave free "verified" checkmarks to people he deemed important to the platform.

Our team remains committed to supporting our communities and our moderators, and we appreciate everything you do for your communities.

i honestly don't believe either of those statements.

I don't understand why we keep getting promised things we don't need and didn't ask for while simultaneously having actually useful things taken away. It's honestly really frustrating. The spam category in mod tools being removed was not a benefit. The app is an absolute joke for trying to moderate and even when it works as intended, its just not great.

I see different suggestions and ideas all the time in the mod sub. I think we should be able to directly modmail users from their username drop down list. That would be a helpful tool.

I don't understand why not notifying users about why their content is being removed would be beneficial to anybody. Unless I'm misreading that?

Edit. Oh and the whole "don't take a screenshot" prompt is really so obnoxious. Why can't we take screenshots?

I’ve spent the last couple of “spare time” years working on a handful of AI projects — one focusing on detection of ban evasion from submission characteristics, with varied success; and a second exploring automated detection and classification of harmful bots, with much more considerable success. The legal changes wrt AI mean I can’t continue that work. That’s fine. It sucks, but I still have everything I learned and the insights I gained.

On the other hand, killing Pushshift actively makes my life as a mod infinitely harder. I can’t cross-check claims in mod mail. I can’t see why people are commenting similar sentiments on a deleted post. I can’t get a feel for a users posting patterns when deciding if they are harmful to my communities.

For all of the shit stunts Reddit has pulled over the years, this is the one that hurts the most. You are actively harming the literal army of volunteers that keep communities running smoothly. For me, at least, eventually I have to make a cost-benefit decision on whether or not curating some/all of my communities continues to be something that will bring me satisfaction and purpose.

I think the answer to that question is no, it won’t.

available to help

we have decided to revoke Pushshift’s Data API access beginning today

Pushshift is reddit history, which is one of the things that keeps users loyal to the plataform

Brave bet. not helpful at all for mods, users or having healthy communities

I’m not going to lie, PushShift was super useful for moderating and this change seems very over the top.

Having access to deleted and removed comments is especially important, for example when a user deletes their own rule-breaking comment or when Reddit removes a comment, as when either of those are the case, the moderators cannot see what the comment originally said, which was where PushShift came in.

Are you guys just trying to speedrun how fast you can run this site into the ground?

• The horrible redesign
• The garbage first-party app
• The amount of spam bots, repost bots, and comment stealing bots that the community has had to try to fight against.
• Allowing garbage ads like "he gets us"
• removal of archival bots

What's next? IAP to buy upvotes and downvotes? This site feels like it's completely lost its roots. Aaron Swartz deserves better.