all of my base are blocked by the escapist :(

[vdrandom]

Hello, dear Escapist.

I've been a member of the Publisher's Club for a year and a half and enjoyed it very much so far. The news, the articles and even though LRR, Jim and MovieBob have left the Escapist, there is still enough content for me to keep supporting it. One of the reasons I have joined it for was the ability to read full RSS feeds. But that ability was shut down for me recently. No, the link for a full RSS feed is still there. The problem is that all of my servers are blocked from accessing the escapist's servers. I run a tinytinyrss instance on one of them (Hetzner, Germany) and for this year and a half had to use an DigitalOcean droplet in Netherlands as a proxy, but not anymore.

For some reason you have blanket blocked Hetzner before I joined you, Escapist, and now you block Digital Ocean (I also fail to connect from another hosting provider, the one I work for). And frankly, out of my 100 subscriptions, you are the only one who does it. As a sysadmin, I'm quite surprised with how you, a news network and a content hub, handle defending your servers. From both ddos attacks and readers. I'm fairly sure I'm not the only one shot down that way by you. And I ask you and your sysadmins to be more professional and more competent than that.

I cannot traceroute after te2-5.dist01.ral.peak10.net, it seems the connections are either filtered or shielded there from tracing, but I do reach that server from all of my servers as well as my home pc I'm writing this post from. Sadly, only my home pc is allowed past that.

I don't have much faith in the Escapist reacting to that message, so in the meantime, if any of you fellow Escapist users are aware of any worthy media and gaming related news networks, please let me know. I have already subscribed on RPS, but as far as I can tell they focus mostly on videogames. That's what I like about the Escapist among other things: the diversity.

--
TL;DR the Escapist has blocked all of my possible means of following their news. As a member of the Publisher's club and a supporter of the Escapist, I am disappoint by that. And looking to reach the Escapist about this / for alternatives to the Escapist.

 

[IceForce]

And I ask you and your sysadmins to be more professional and more competent than that.

Everyone is going to tell you the same thing: the "sysadmin" doesn't read these forums, so they won't ever see this post.

You're asking in the wrong place.

 

[DoPo]

I don't have much faith in the Escapist reacting to that message

Staff don't lurk that much in the forums, but if you write to them directly [http://www.escapistmagazine.com/contact/] you should get a response.

 

[vdrandom]

Well, a while ago, when I first joined, I have tried contacting them on that issue using that very form. To no avail, sadly. That's when I've started using a proxy server in DigitalOcean.

Thank you anyway, I will make another attempt.

 

[Kross]

And I ask you and your sysadmins to be more professional and more competent than that.

Everyone is going to tell you the same thing: the "sysadmin" doesn't read these forums, so they won't ever see this post.

http://rss.escapistmagazine.com/ is a mirror through our CDN, and nothing is blocked from accessing it. The old /rss links should all redirect to this domain, and have for several months now. Specifically for people who are self hosting RSS aggregators like this.

I also have poked several holes through the firewall for people who have let us know [http://www.escapistmagazine.com/contact/] what their static IP is. I'm sorry if your prior ticket was lost in the shuffle; we do follow up on everything, so there may have been an email issue.

Data centers are blocked from the web servers due to rampant vulnerability scanning and scraping/crawling (and heavily abused anonymous proxies used by spammers/etc here on the forums) that are using resources or otherwise doing things we don't want to pay for or babysit.

There is extremely little legitimate website traffic from data centers (all the major crawlers that send traffic like google/facebook/etc have their own IP allocations), and what little there is very rarely view ads that pay for their resource usage. And for the remaining few that do need proxies for actual connectivity reasons, I am always happy to work out a solution.

Sorry for the inconvenience.

 

[IceForce]

the "sysadmin" doesn't read these forums, so they won't ever see this post.

And then the sysadmin responds to this thread.

Well, that's made me look like a right tit, hasn't it...

 

[Kross]

the "sysadmin" doesn't read these forums, so they won't ever see this post.

And then the sysadmin responds to this thread.

Well, that's made me look like a right tit, hasn't it...

I regularly search for people complaining about connectivity issues.

 

[Kross]

The only true statement there is that people with the knowledge to protect their traffic will probably not be seeing ads. The number of people protecting themselves however is in a literal geometric progression. You think everything just went https compliant for fun? Some of us don't like to be sold as products, or spied on by paranoid national security apparatuses.

I do love being a lying liar!

Anonymous proxies have been around forever, though people like to call them "VPN"s now.

I'm currently working on rolling out https, but waiting on upstream to enable it for the CDN. I'd also love to work out a way for users to be issued client certs that would allow me to give them a special server that's not blocked from anywhere other then the need to have been issued a certificate via your account.

I don't like having to make a living off of website ads any more then anyone likes to view or be tracked by them, but it's unfortunately the revenue stream that pays for my food/clothes/rent/insurance, so ensuring that festering resource goes as far as possible is part of what I have had to come to terms with (along with mitigating the many occasions aggressive crawlers have lagged the site, or spammers have burnt out our volunteer moderation team / otherwise shit up the discussion areas of the site for the general community). No ads from pubclub are a compromise for our audience, but are a long ways off from reaching the critical mass of allowing a full time staff to live off of them (never mind the interests of the larger company that is keeping everything online in return for being able to include our demographic in their larger portfolio's revenue stream).

Edit: Apologies in advance for blocking your current proxy. :/

 

[vdrandom]

http://rss.escapistmagazine.com/ is a mirror through our CDN, and nothing is blocked from accessing it. The old /rss links should all redirect to this domain, and have for several months now. Specifically for people who are self hosting RSS aggregators like this.

Interesting, thank you very much. It probably didn't work because for my proxy I had to download the feed first and then grab it from my proxy server's http (cronjob with wget, so no caching). Until it was blocked today, that is.

I also have poked several holes through the firewall for people who have let us know [http://www.escapistmagazine.com/contact/] what their static IP is. I'm sorry if your prior ticket was lost in the shuffle; we do follow up on everything, so there may have been an email issue.

Wonderful, although I'm not really sure if dealing with each proxy personally is that efficient, but I'll let you be the judge of it. I'll also provide you with my proxy's ip address. A suggestion: using cross-checking direct/reverse dns records would be an easier way to automate this if you happen to try and do it at some point. No bot/free proxy will care enough to make their dns records match before scanning you.

Data centers are blocked from the web servers due to rampant vulnerability scanning and scraping/crawling (and heavily abused anonymous proxies used by spammers/etc here on the forums) that are using resources or otherwise doing things we don't want to pay for or babysit.

Why not react based on the amount of requests per second from specific addresses? Too resource heavy/inflexible for your needs? Are there are that many vulnerabilities on your site? Why not only limit access to the sections vulnerable to spam bots, like forum pages and login form?

There is extremely little legitimate website traffic from data centers (all the major crawlers that send traffic like google/facebook/etc have their own IP allocations), and what little there is very rarely view ads that pay for their resource usage. And for the remaining few that do need proxies for actual connectivity reasons, I am always happy to work out a solution.

Do you block datacenters by their networks? Do you make sure it is a datacenter before blocking it?

Sorry for the inconvenience.

I should also apologize for that line about competence. Was a bit frustrated. Still don't have enough info to properly question it so count that out.

EDIT:

I'm currently working on rolling out https, but waiting on upstream to enable it for the CDN. I'd also love to work out a way for users to be issued client certs that would allow me to give them a special server that's not blocked from anywhere other then the need to have been issued a certificate via your account.

Yes please! Authenticating with a client certificate would be so very nice!

 

[Kross]

Why not react based on the amount of requests per second from specific addresses? Too resource heavy/inflexible for your needs? Are there are that many vulnerabilities on your site? Why not only limit access to the sections vulnerable to spam bots, like forum pages and login form?

Do you block datacenters by their networks? Do you make sure it is a datacenter before blocking it?

I've had iptables connection tracking buckets get overwhelmed from traffic flooding (luckily DDoS's are less frequent these days, and tend to be more on the page refresh application load end rather then pure network silliness due to more aggressive upstream filtering).

With the advent of cheap virtual hosting, it is more futile then ever to re-actively block and report hostile traffic. And I've never been comfortable with the false positives of automated blocks or someone else's lists. Therefore when I see aggressive crawling/scanning, or otherwise notice a data center IP, I look up their allocation and block the entire thing with a "dumb" rule to minimize overwhelming the local NIC driver/CPU.

The closest thing to an actual false positive I've seen is that occasionally a College/University will peer through a local data center and not have their own allocation. Luckily I tend to find out about such things pretty quickly.

Unfortunately, with only a half rack of servers and a single administrator, the service degradation and resource usage to handle the level of aggressive traffic constantly hammering any public IP makes this approach more desirable then perhaps my inner information-wants-to-be-freeeee nerd would like. And then when I add in the "what is paying the thousands of dollars a month to keep this online and us all eating food"... blocking huge swaths of non-revenue generating traffic (while simultaneously cutting down on abusive forum accounts that love proxies and have to be manually swatted by volunteers) adds even more benefit.

As I said though, I am looking for ways to open up our accessibility wherever possible as time and creativity allow, and am currently working on a few DMZ-style tricks (like client certs) as I'm able to acquire more hardware to handle the load such things bring. HTTPS in particular was a rough thing to fit in our old setup, as we were already tight on CPU and the overhead was unacceptable for quite a while. A couple small hardware upgrades (thanks corporate overlords) and a lot of industry work on the resource demands of encryption have finally made that possible, and there should be some nice changes in the near future.

 

[Kross]

In short, why not do all of the things that all of the other sites use, and seem to flourish? It's a good question. I have a hard time believing that major news sites, blogs, and the big game sites have these struggles. They have more people, and presumably more spammers too.

Many of them throw more hardware, people, and money at it then I have at my disposal. Others rely on third party handling for resource intensive things like discussion forums/categorization, or don't have such things at all, or use more restrictive gating like forum paywalls to handle spammers, or use third party content hosting like youtube that pays an extremely small percentage for your work. There's a reason that you can count "major" gaming sites practically on one hand, and it's the cost of hosting for any period of time while also paying for full time staff.

The other large content sites tend to be networks like Gawker and Buzzfeed for a similar reason.

There's also a reason why PA Report (along with their hugely successful, but oddly only one year no-ads donation experiment) and Polygon's highly produced reporting only lasted a short time, and it's because of how unsustainable the revenue stream is compared to the resource expenditure. :/

The rest of the ecosystem tends to be single feed Wordpress sites, which often throw massive amounts of hardware at things for scaling (or are static enough to just serve through a CDN mirror), and ignore most things that would deviate from the core functionality.

 

[Kross]

I would love to have radio style bespoke ads from our content creators as our primary way of selling ads.

The other route (well, a combination is best of course) is having a robust enough store to pay for things, but that's a whole other business that we've only dipped our toes in. Donations are really rough because it's a recurring cost, so while they certainly help keep the literal lights on, they still pale in relation to what proper ads (not Google's bulk filler) bring in per month to pay for the human/content costs. The only real way for a direct pay system to work is if they're paying for exclusive *good* content, but that means gating off content from those who aren't paying - and we neither want to limit/alienate our audience or have the critical mass of exclusive content to sustain such a model.

 

[lunavixen]

I regularly search for people complaining about connectivity issues.

I have a problem with the video player, the ad starts playing, but so does the video, and I can't close the ad, it sucks.