SQL injection is an old and known threat, it involves inputting SQL into a form field, or passing it as part of a querystring, with the aim of modifying the SQL statement itself. Whereas normally the input would form part of the criteria in the executed statement, an SQL injection attack changes the nature of the SQL statement itself which is executed in the website script.  Not a good thing!

Historically such attacks have been relatively easy to prevent.  Good validation of input on the client and server allows the syntax required to create such an attack to be intercepted, thereby rendering it harmless.  Any web designer worth their salt would have in place one or more routines that checked submitted input, before allowing it to become part of the SQL executed in their application, so it was a manageable threat.

Then came ASProx, a new form on injection attack to which ASP scripts and MS SQL Server where particularly vulnerable; it could bypass established validation routines and “appear” to be valid input, but once executed as part of your SQL script, could do pretty much what it wanted with your database, and usually your web pages built from data in your database.

A couple of things set this apart from traditional injection attacks.

• It was automated. The attack was delivered by a Bot, it searched for potentially vulnerable ASP pages, and would run until the payload (malicious code) was detected on the webpages. The Bot would then move on to the next website.  The big deal here is that the Bot would come back, day after day, so even if you cleaned your database up, without implementing some further protection in your script your database would be compromised again and again and again.
• The attacks used Fast Fluxing, which basically means the source of the attack is difficult to identify as the Bot uses the infected PCs as proxies.
• The syntax of of the injection attack was unusual.  It was encoded, so that quite a long SQL script could be passed in the querystring, but the usual injection checks could miss it.

I wrote a basic function that will test for the existence of a couple of ASProx signature traits - it’s ideal to wrap around your database connection code/include as if ASProx can’t connect it cannot do any damage!

I’ve provided the syntax below as I noticed some people are charging for similar help!!!!!!!!!!!!!!!!!

Function noASProx(thequerystring)
 ” Querystring injection check for ASPROX
  strquery=thequerystring
  strquery = Replace(URLDecode(strquery), ” “, “”)
  If Not InStr(UCase(strquery),”EXEC(”) > 0 OR Len(strquery) > 500 Then
    noASProx=1 ” clean string
  Else
    noASProx=0 ” infected string
  End If
End Function

The full querystring needs to be passed to this function using the request.servervariables (”querystring”) code.  If True/1 is returned allow access to your database connection code/include, if it’s not don’t!

If you’ve already been hit by ASProx, put the above in place. Then I’d recommend connecting to your SQL Server Database using an MS Access Project. In doing so, you can, without needing to write a complicated stored procedure, perform a “find and replace” on each table in your database.  Obviously you find the malicious payload code which (normally) starts with <script> and ends with </script>, and you replace with nothing - thereby deleting it.

The people behind ASProx are clever - they found a weakness, they exploited it. I don’t condone it, but I often marvel at why people with intellect like these guys obviously have, choose to use it in this way. I wish I had their talent.  Legitimately applied to what I do - I’d be a millionaire!

Talking to other small business owners I’ve met, few, er well none actually,  would cite the December 1st 2008 VAT reduction as a good thing.  It has cost small businesses money to administer and account for the change.  At both eantics and IFA Portals we chose to implement the change and make sure that we passed the rate reductions back to clients - most of whom pay monthly and aren’t VAT registered so can’t claim back the input tax.  We’re not talking about a fortune, 2.5% of the average net monthly invoice is not much - but it was the right thing to do so we did it.

However, the costs involved in time and postage e.t.c are proving to be significant, not to mention the opportunity cost - how we might have spent the time otherwise i.e. marketing, on the phone to clients, pitching for new business. 

Our clients are mostly paying via standing order, which we can’t control, but IFA Portals has a new direct debit facility so we wrote out to all our clients explaining the change, and that we needed the direct debit mandate back.  We also mentioned the standing order needed cancelling by them - as we can’t do it.

So we get the direct debits back and calculate their first payment for January which is a curious mix of VAT calculated at 15% for the month ahead, a rebate for the  month of December as they paid VAT at 17.5% (there was no way we could respond by December 1st 2008), and it’s prorata as the direct debits are collected on the 1st of each month but standing orders fell whenever their bank set them up.  Once we knew the charge we wrote to them to inform them as you must with direct debits.  We need to write to them again to say what their regular payment from February 1st will be, but that aside, job done we thought.

Not quite, we’re taking a lot of calls and emails querying the new amounts as they are not the usual amount, and in some cases taking calls and emails from people accusing us of charging them twice! Yes, you guessed it, they didn’t cancel their standing order despite the instructions we communicated, so now we’re into a round of issuing cheques for the overpayments and the associated bank costs that go with that. 

I absolutely understand why people have forgetten to cancel their standing orders; it’s Christmas for one, but more importantly the financial services sector is badly hit by the current economic state - our clients have more important things on their mind.  So we’ll refund all the overpayments promptly for the same reason we thought it important to pass on the VAT reduction in the first place.

I anticipate much more of the same for the rest of January, and I’m left wondering what Darling thought he would achieve by reducing the rate of VAT.  Businesses like ourselves and IFA Portals that bill on a monthly basis have incurred significant costs and time out from the day job, at exactly the time when small businesses access to cash (credit) is more restricted than ever. Small businesses prospects for acquiring much new business in the short term look average to poor, which mean small business owners must work harder at marketing and client acquistion (and retention!!!!), but instead they’re messing about with VAT reductions!

Not a good recipe for kickstarting the economy in my book.

Not all websites are set up in such a way that you can select just the data on the page you want.  Some will only let you grab all the page, and some frustratingly appear to let you grab the page but don’t return anything in your query.

If you need that data for use in your Excel application this is less than satisfactory, so it’s time to play hardball and leverage the power of VBScript with your own home grown scraper.

I’m not going to do a step by step, as anyone reading this is probably able to grab the concepts and do it themselves, so here are the fundamentals.

You may have heard of Ajax.  In windows ASP applications the engine room of ajax is the XMLHTTP Object and it’s this object that we will use to power our scraper.  This ActiveX object will grab an entire page of HTML and hold it in memory. 

All you need to do is work out how to parse that data to get at the bits of data you need. This is where the split function comes in, which will split the data into an array based on a separator you choose. The first split you might want to make would split the page into rows and you do this by splitting on chr(10).

mynewarray = split(mypage,chr(10))

You then need to work with that array, a good first step is to learn how big it is

mynewarraysize = ubound(mynewarray)

The array starts from 0 when created via the split function, so if 12 is returned you know your array has 13 elements.

So what now.  Well you could output this into a worksheet, using a loop based on the array size to write to cells, or you could perform further split functions to get at the data you want.  You might also use replace, to strip HTML tags in the data that you don’t need outputted as part of your information.

I might post some more on this at some point, if people want to know more.

The good news is that if a website is designed to W3 standards both for it’s HTML markup and CSS markup, it is already a long way down the road to being accessible.  Here’s a useful tool you can use to evaluate just how accessible your website is.

The application has come on a bit since then, I’ve basically changed it so it searches on main towns; most doorway pages are created on main town names rather than place names which could include hamlets and villages.  This takes a database of 26,000 down to just under a 1000 and makes the load on my servers more bearable. The downside is it doesn’t show just how spammy some websites are!  Believe me, some have a doorway page created for everyone of those 26000 based on the results the spider returned!

It’s now also progressed beyond spider - it will be a web application you can use in the next few hours.  It’s quite simple - and rough at the edges,  but I intend to post the code and the database for others who want to build something like this or help make it better.  It’s written in ASP and uses a MySQL database on my server (now that’s what you call a mashup) - I’ve named it oopsydoorway.  You can see the doorway page detection application here

Current issues

• Badly configured websites.  If a page does not exist, a 404 error should be returned.  This would tell the application there was no page there.  But many websites (particularly blog apps) it seems return a 200 Ok  and not a 404 Not Found response.  This gets them listed as a potential doorway page,when they might not be, as the server might not be configured correctly.  On the other hand they might; returning a 200 Ok response is a known spam technique.
• Directories.  Of all the websites that might have a valid reason to serve up geographically tailored URIs, directories are probably top of the list.  Is it spam or is it good categorisation for the purposes of indexing their results? To be fair - probably the latter.  But Jo Smo Webhost, based in Accrington and working from his bedroom, isn’t running a directory so his doorway pages are spam.
• Encoded URLs - It’s missing a few don’t know why at the moment - will look when I get a moment.

I’ll post the code and the database data once I’ve got the app live and made few tweaks.  I’d like the app to be able to score a website, i.e if it returns 2 geographically tailored URIs its hardly criminal, but 10, 15 or 20 that’s a different matter.  I’d also like the app to be able to complete the Google spam report.  I don’t think this can be done via their API, so might have to settle for cut and paste.

My stance is DON’T go that route  - instead build a better resource, write more unique and high quality content;  it’ll get the links and give the search engines something they can’t find elsewhere.

But others in my sector, who are building their web marketing strategy around doorway pages (and given my “sector,”  they are probably doing it for their clients too)  are,  in many cases,  ranking higher than my site!  It doesn’t lend much credence to my theories - not in my Clients’ eyes I can tell you!

I’ve submitted spam reports in the past, but in my opinion a problem with them is that you don’t get any feedback.  Certainly, the sites I’ve reported are still in the results, where they were before, as far as I can see, so at best I’m left assuming that my report contributed to the algorthym,  or at worst wondering if anyone acted on it, or even saw it.

So, I’ve just got hold of a database with 26000+  UK place names in it.   But what to do with it?

Well, I could punch out 26000 different doorway pages (and the rest) for my web design website and probably get much more traffic and enquiries than I do now. They say, if you can’t beat them join them, but I’ve no plans to.  I could more likely get bounced out of Google knowing my luck!!

Instead, I thought I’d tackle the problem head on.  Given than Google can’t identify doorway pages to my satisfaction, I thought I’d have a bash, my ultimate aim - a simple website that allowed anyone to search in their business and geographic sector, identify the spam results, and submit spam reports to Google.  

So far I have built a three spiders:

The first looked for links to doorway pages.  That didn’t fly. Doorway pages often don’t have any link out to them. The only links are inbound to the homepage.  This renders spidering ineffective as a means to determine the extent of the doorway pages.

Next I went straight to the sitemap. Another blank. Doorway pages, would you believe, often don’t appear in the websites sitemap.  So there are no clues in there either.

This is not easy…. change of tack I think.

Actually, my third spider works! It uses that database of 26000+ UK place names I mentioned earlier, it basically picks out geographically tailored URLs (at the moment based on a seed), and then looks for similar pages using place name substitution in that pages URL.  i.e take a search for “web design warrington”. It would check the database, determine that warrington is in Cheshire, then test that same page for all other Cheshire place names.

You can spot the problem? Other than it being resource intensive on my servers, it’s quite unfair to websites who are not employing spammy optimisation techniques - if they feature the seed (”warrington” in the above case) in their domain, directory or file names, they’ll be fed just as many URLs as the spammy ones, but they’ll return 404’s instead of 200 responses. Not Ideal!

However, so far I have uncovered a couple of websites that are absolutely taking the michael.   Yes if your logs show 200’s on all your pages in a 3 minute time frame - it was me.   Love the fake 404 pages that are return 200’s as if someone messed up the server configuration - very nice though if you mistype the placename - you get a proper 404! 

I’m onto something though, it needs refining, it needs a way to determine the most likely placenames that will be used to spam based on the location already found, some sort of distance calculation, then I think It’d be fairer to the clean websites as it could give up after the obvious keywords had been tried, thereby not filling their logs with irrelevant 404s and it wouldn’t take my server away from me for minutes at a time!

I’m going to keep at this and I’d love to hear about any similar attempts to build spam busting spiders too.