Re: Can't Pull Pages From External Sites

From: Joe Kaplan \(MVP - ADSI\) (
Date: 09/09/04

Date: Wed, 8 Sep 2004 22:12:10 -0500

You probably need to send a cookie in your request as well that will
authenticate you. This would be the same cookie that is sent when you visit
the site with your browser. Otherwise, you need to negotiate the login
pages programmatically.

Joe K.

"RL" <> wrote in message
> Hi,
> I'm trying to pull pages from news sites like New York Times and WSJ. (I
> have accounts with them.) I wrote a ultility that goes to their home
> pages
> and pulls out the links that I want. But when I try to get those links,
> authentication fails--I get login pages instead (I don't get those login
> pages when I access the desired pages from a browser). I tried to set
> request headers to make the request look like it's coming from IE, to no
> avail.
> Any ideas would be appreciated.

Relevant Pages

  • Re: How to share session with IE
    ... my browser module if necessary. ... program can load the cookies from your real browser's cookie store ... "need to login" condition, and react accordingly. ... Another option instead of making your program run through a series of clicks and text inputs, which is difficult to program, is to browse the html source until you find the name of the script that processes the login, and use python to request the page with the necessary form fields encoded in the request. ...
  • Re: set cookie in nusoap web service, IE behaves diff than Firefox
    ... > browser as the first output. ... > Works fine in IE6 and the service returns the state of the cookie in the ... it rather implies that $this->headers refers to the headers sent ... I don't think 'Content-Type' is required in the request. ...
  • Re: session riding
    ... > When a user browses my script I'd like to grab a session cookie from the ... A normal browser will only send you cookies in the same ... domain as the request, so this is likely not possible. ...
  • RE: HTTPModule - an interceptor indeed, but without communication skills!
    ... httpModule to check in the certain event before request has been processed ... easily manually append such querystring to bypass the validation. ... My suggestion is what about the cookie? ... In the validation code, you can use javascript ...
  • Re: Printing website - cookies
    ... would be requested with the Print command so that the latest version of any ... gets that cookie). ... At the request of any "private" article or image, ... All "private" images in the article turn out, on paper, as the default ...