MAC OS X, Linux, Windows and other IT Tips and Tricks

04 May 15 Using CURL for sending crafted HTTP POST authenticated queries

I came across a situation where I needed to send an HTTP request using the POST method with some POST data but after I have authenticated with name and password.

SOLUTION:(using curl tool)
The trick here is to preserve the SESSIONID of the authenticated response for the second POST request.

I needed to go into my account in and request the CSV file which lists all my registered domains.

curl -v --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0" -c cookies.txt -d "username=myuser&password={html_encoded_password}"
curl -v --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0" -b cookies.txt -d "orderField=&orderDir=&name=&state=&owner=&sedo=&lock=&date_expire=&renewal=&itemsPerPage=&csv=CSV"

In the above example I simulate a Firefox Browser (–user-agent), save the cookies (includes the SESSIONID) in the file cookies.txt and use it in my second POST request to get the content of the requested CSV file into the terminal.

IMPORTANT NOTE: The password must be in proper HTML-encoded format to be accepted. This is applicable for any chars. that is not a-z or A-Z. There are many ways to convert the password in HTML-Encoded format. The most reliable way I found, is to manually login with a proper browser with name and password and look at the request headers using a browser plugin that lets you see the headers contents. The password will then be shown properly in the header.
Examples of password characters and their HTML-Encoded equivalents:
& = %26
! = %21, etc.
So a password like: Tw&Ui8vH!
would look like this: Tw%26Ui8vH%21

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: