Half of the posts are keyloggers, questionable scrapers, social media bots and auto-clickers, replacement programs for LONG solved problems with slightly suspect code, etc.
I know python is easy to learn and all, but is this much of the python community really just building scripts for semi-questionable software?
Edit: I almost forgot the "do my homework" posts.
74 comments · 67% upvoted · other discussions
My guess is people like sharing whatever they learn to do. Also they see other people doing it and that gives them more confidence to do it. Its a vicious circle.
Also, I'd argue most people that come here to learn python are not studying something in IT (talking university), and they are just in other line of work.
I think that the intersection between python users and reddit users is that. I'm a professional developer, but like python for small, personal projects, that sometimes just end up being some sketchy script. I don't post it though
Python really kind of is one of "data science" languages. There are more, just can't remember. That's not the point though.
The point is that scraping really is a good way to get data. Python does this really well. It's also really good at having a simple language to parse through saved data.
I'm not sure why theres so much sketchy stuff, but I dont believe that's all of it. I use python to make APIs as well as scrapers. It's all about preference and project. Some people are into keyloggers for whatever reason.
I personally am into data. I got into this career path to manipulate, share and collect data (reasonably). So most of my projects have to do with some kind of data.
python is popular with security types for the same reason its popular elsewhere - rapid development of simple stuff
I don't think it's all of it by any means. I use python for all kinds of DevOps tasks. However, I find that these types of things are the majority of the posts here.
I wouldn't know if that's specifically true. Do you know how you could find out... other than the reddit api
Why wouldn't you use the reddit API?
I mean I would. Because it's easier. But you could scrape it.
Right but why would you scrape it?
Reddit rate limits its API, so if you want “big data”, you can’t use it. People are interested in big data projects because small data personal projects aren’t as impressive/relevant if you are applying to big data companies.
So when the API prohibits pulling the information at the scale wanted, then scrape the content instead?
I’ve seen it done that way, yes. I’m not sure, but it seems the Reddit API is just a courtesy, not an indication of the only data people are allowed to access programmatically from Reddit. Never have done it myself, though.
There are other limitations to the API, like you can’t get at a lot of historical data. This is all second hand info. Just trying to think of reasons people don’t necessarily care to use the Reddit API.
Do most web scrapers take robots.txt into account?
Most people are probably beginners so most of them won't even know robots.txt even exists.
It should be mentioned more in scraping tutorials etc. I feel like a lot of scraping is potentially just stealing content.
I know several places I wanted to grab some info from had policies prohibiting it. I'm fairly curious to know if it is a standard practice to check into robots.txt and a websites policies before scraping and using their content.
In the case of for-profit projects, is it common for scrapers to ask permission before profiting from the content scraped?
Sure you want to check the robots.txt and policies. If the site owners list a lot of restrictions you know they may make scraping difficult and you should build random delays and user agent switches into your code from the start 😀
It is almost always the professional and personal thing to do to contact a site owner before you scrape it.
Do I? Sometimes. Depends on the situation. I work in news and build a lot of scrapers. I make a point never to hammer a site. I sometimes ask permission.
A lot of times I'm getting government info - it's mine already and I'm mainly scraping it because it's easier or faster than dealing with a records request, or because the agency has tried to fuck me around.
The other time IDGAF about permissions is when I built scrapers for event listings. The companies involved in that were mostly built on scraping and aggregating listings. I give zero fucks about returning the favor.
What are some examples of 'for profit' ethically questionable scrapers you've seen?
I wrote one myself that I was questioning of, that reached out to the SEC for a pile of records of certain types of fillings. While it was govt. info, I'm not entirely sure I was supposed to be abusing it that way. Technically I was 'reworking gov data', but...something didn't feel exactly on.
Have had my fair share of idiots wanting to scrape their facebook channel to populate their website with, etc.. It's like NOBODY wants to hear you say API.
Pulling SEC fillings is kinda the definition of public record. I would feel fine with that - again, those are both legally and morally your records already. I'm surprised they don't offer an FTP download. What didn't feel OK to you?
Facebook only offers an API until they can corner whatever vertical they're in, then they kill it or monitize it. FB is the enemy of the web.
So basically you'll do what you want regardless of policy or whatever. Moreso if the policy is in the way of what you are trying to accomplish.
Ran across some questions a few weeks back about building a real time scraper of craigslist and FB marketplace to create a combo marketplace comprised of the posts from both places.
What do you mean policy? When it comes to public records and freedom of information I'm pretty close to an absolutest. I'll make an effort to be courteous with government agencies because I'm not an asshole and I'd rather not waste time on lawsuits, but I have zero tolerance for BS.
With private organizations, I'm a bit more nuanced. I made sure the local performing arts center knows I scrape their events listings once every couple of days.
Same with a dozen or so other venues that don't really manage their own sites (WordPress built by contractor mostly. They can't spell API). None of them would know the difference is I hadn't.
I would probably be fine with scraping anything from FB because they're evil, and Craigslist because I've got a personal grudge, but the point is that I don't do whatever I want, I make a decision about the ethics.
How is it stealing content if it's content that is already public. In the example of zillow or trovit, all their real estate information is already there. It's public and anyone can see it.
That's like saying, "you can't gather dead leaves off the ground". That said. Zillow has not detection and blocks bots. That's cool, I don't feel like programming a work around to your captcha. You win.
How is me scrapping a real estate site any different than me spending hours going page by page and manually entering it in to a spreadsheet?
Many websites that provide information of one form or another explicitly state that the content cannot be harvested and used commercially. And it is their legal prerogative to do so and is enforceable in most instances.
For your zillow example, their policy states:
You may not utilize Zillow Data or distribute Zillow Data on a mobile app such that Zillow Data is the primary functionality on the mobile app and/or Zillow Data makes up the majority of the content on the mobile app. You may present Zillow Data only on a transactional basis. You will not permit your users to access the Zillow Data in bulk. You may not retain any copies of the Zillow Data. Your license to Zillow Data is limited to making direct server calls to the Zillow API for the Zillow Data and distributing the Zillow Data to your end user(s) on your Site(s), immediately upon receipt by your servers. You may not present the Zillow Data (or permit the Zillow Data to be presented) so that it appears to be available from a third party website. You may not use the Zillow Data for direct marketing (i.e., mailing) or telemarketing lists. You may not separate address information from the Zestimate® or Rent Zestimate valuation for the property at that address or the for-sale or for-rent information, if applicable. You may not separately extract and provide or otherwise use data elements from the Zillow Data to enhance the data files of third parties.
So, grabbing a bunch of zillow data with python outside of the API is explicitly forbidden. Copying it by hand and storing it is explicitly forbidden. Grabbing a bunch of data, breaking out what you need from it, and presenting what you need for your own uses is against the policy.
Just cause something is accessible to the public doesn't make it free to use however you please.
Your anecdotes are bad. Especially your dead leaves example. It's explicitly illegal to go to a public state park and take bags of leaves and bundles of sticks home.
Next to zero content available on public television is legal to copy and rebroadcast for anything but personal use, let alone for profit. Same for a lot of website content.
Edit: To answer your question directly
How is me scrapping a real estate site any different than me spending hours going page by page and manually entering it in to a spreadsheet?
They aren't. Both are against the use policy.
it is their legal prerogative to do so and is enforceable in most instances.
INAL, but I'm pretty sure you're contentions are either mistaken or irrelevant.
There have been a number of cases over this ground. A site's TOS doesn't carry much if any weight in law to date.
In the federal 9th circuit court of appeals right now LinkedIn is trying to get permission to sue some company that's been scraping them. LinkedIn is trying to say that by totally ignoring the TOS the scraper is breaking the CFAA. The reason it's at the appellate level is because the trial court judge shot it down. It's been heard by a three judge panel but no decision yet. From what I've read the judges who heard it were pretty skeptical of LinkedIn's position.
A decision in LinkedIn's favor will break with at least two earlier cases I'm familiar with and several others that I've heard of but don't know well. I don't like their chances, unless they can just outspend the other company into a settlement.
Next to zero content available on public television is legal to copy and rebroadcast for anything but personal use, let alone for profit. Same for a lot of website content.
Yeah, except under U.S. law data is not copyrightable. Copyright lawyers call it 'database rights' IIRC, and it's black-letter law that there are none in the U.S. The EU is different and I don't know the law there well, but in the U.S. the TV analogy is almost as bad as the raking up leaves from the national forest analogy.
Reality disagrees. A quick google shows that:
All of which won due to ToS contents or copyright claim to the way in which the data was arranged or aggregated on the suing site.
Interesting... in either case I dont care? I'm not selling it or releasing it so I dont really care what they want. If they want it to not be obtainable, hide it.
That's what I expected when I asked. Based on the types of comments I see in scraping posts it feels like most scrapers are not acting in good faith relative to data policies.
[deleted]
I was quoting legal cases in relevant to the discussion in that thread, i.e. that the ToS were not enforceable and that you can't copyright data. In the cases quoted there - the arrangement of data scraped was considered copyright infringement, or the ToS was enforced by legal means.
Overall what I'm hearing here is that the scraping activity I am asking about doesn't operate in good faith relative to the ToS or legalities involved. Most of that seems to be due to a general disagreement over what can or should be controlled access to data, regardless of what the content owner has to say about it.
Hence the "sketchy scripts" in the post title.
Edit: Also, I didn't make the determinations on those cases, I have been researching lawyers reviews of precedent in the area to see for my self where the grey (read: sketchy) areas lie.
Actually, you know, when I've built scraping/search engines, I tend to use something like Java for the actual scraping, and then Python for the actual analysis. Python certainly makes it easy to get something workable up and running, but bigger systems like Crawler4J and Nutch are much more tailored for the task.
If I need something fast, usually wget does the job in recursive mode.
I think it’s just people who are learning, wanting to show off there progress. I don’t think there’s anything wrong with that just as much as I don’t think there’s anything wrong for someone positing a highly advanced project. This is a python community not a python masters and not a python noobs community, just a python community.
Anyone can post what they want, it’s not like these users are being malicious in anyway on here, just building stuff they think it’s cool.
That’s how I learned! I’m sure you learned the same way :)
I don’t know if this sounds mean to you op, I’m not trying to have any sass or negativity, I just think this answers your question fairly well.
I built a keylogger.. Im QA and I use it while I test manually. Allows me to always be able to look back at my actions. Sometimes something happens and you forget every single detail or get distracted. Helps a bunch. I want to eventually extend my automation to auto-generate a macro based on it.
Special case I know.
It's a good use case I hadn't considered for one honestly. I spend hobby time woodworking and I have a video recording setup out in the shop. It was the leftovers of some old surveillance systems from work.
I don't stream or make youtube videos with it though. 30% of the time I use it to figure out what I did wrong when something get's messed and the other 70% of the time I use it to figure out where the hell I stuck something I can't find.
A lot of my "legitimate" contract work is based on these kinds of ideas.
People often come up with ideas for content aggregation ... which of course involves scraping that perhaps is a little sketchy in many cases.
One of them was a "high end yelp" ... so they wanted to scrape NYTimes reviews and the-like ... it's only really illegal if you break the copy-right laws and publish the whole articles you scrape rather than a tiny portion + link and attribution (which is why I eventually cut that contract loose... lol).
Another I'm working on involves scraping purchase history from a users' online purchases ... at amazon and the like. Sketchy? I guess at first glance ... but if you know the whole project idea ... not at all.
My take is that creating a working library on github/pypi is a 'mandatory step' in becoming an established 'python expert'. It looks good on the resume.
Google crawls millions of websites.
Is someone at Google finding and reading the policy page of each website beforehand? You think their crawlers can interpret the "legalese" of policy pages before crawling and profiting from the data? We both know the answer is No.
Why is it okay for Google to crawl and profit from data on the web, but for some reason it's not okay for people here on this subreddit?
Where?
This is a perfect example:
https://www.reddit.com/r/Python/comments/b0j97i/my_another_video_on_discovering_hidden_apis_on/
Literally a video on stealing higher tier access on paid services.
That's a good example.
What are you the internet Police? Get over it and on with your life.
Ask legitimate question, get snarky reply full of assumptions. Way to go.
It's your attitude in your comments on this shit post.
So the fact that I wonder if all the scraping going on is just content theft is a problem?
You don't like that I question the motives of sketchy posts on reddit?
You don't think that web scrapers should be concerned with content policies on websites?
You don't think I have the right to be interested in how these topics are treated by fellow /r/python users?
And I'm the internet police. You are a silly person.
how is the average scraper any more or less questionable than facebook? zuckerberg is in the uk trying to explain himself to the government again this week, isn't he?
We aren't asking if stealing is better or worse than killing. We are talking about how much "sketchy" stuff shows up in the /r/python subreddit.
Take your FB stuff elsewhere, it isn't relevant to this discussion.
Edit: We already know facebook is sketchy.
It reminds me of the anecdote of a fella who FINALLY made his way into a data science/programming/IT role (paraphrasing) and he was THRILLED to finally peel back the cover and see a gorgeous, flawless interworking of code his new company was built on. But once inside he saw the reality, popsicle sticks held together by deal tape in a massive trembling structure. His job was constant maintenance, debugging, and identifying next steps for the business.
I supposed the lesson here is, this is not an exact science, mistakes teach faster that perfection, and we can’t let anyone know it’s all just a huge unstable house of sticks we are slowly building to stability
realboabab 5 points · Feb 2019
lol, I think this is the standard experience rather than mere anecdote. back to breakpoint debugging through local code running against live databases for me...
Log2 3 points · Feb 2019
For any programming language, might I add.
c_o_r_b_a 2 points · Feb 2019
I'd be interested to know about someone who joined a company and didn't have this experience.
sh0rtwave 2 points · Feb 2019
Or you see interesting emails, like the one I used to get that showed me: Tables with high identity values.
Just think that one through, and then WHY would I need to know...that?
__deerlord__ 1 points · Feb 2019
Integer overflow maybe?
sh0rtwave 1 points · Feb 2019
Right. On a *table ID column*.
Basically, this means the database has some tables that have almost reached the upper limit for automatically generated record IDs. MS SQL Server. This thing had to have some dude from MS show up weekly to keep it tuned, it was so packed with data.
__deerlord__ 1 points · Feb 2019
Well that makes sense that /someone/ is going to get notified. We've had sequence issues on some of ID columns, its...fun
sh0rtwave 1 points · Feb 2019
The crazy about it, was literally that: "If we don't keep an eye on it, a half-intense sales surge will kill the whole thing".
If it were me, I'd have stashed everything older than a couple of years into some kinda fast, queryable archive (even if it was just another table. Joins make life easier in this regard...)
scooerp 1 points · Feb 2019
Reaching overflow for database id isn't necessarily the sign of bad code or poor design, but just a large system that's been running for a while.