PreviousNext…

Site revealed: simple referrers

Background

I read a lot of blogs, and I particularly like some of them done with the big players in blog software, like Radio UserLand. One of the things many of these blog sites have is a "referrer log", a pane of anchor links to other sites, generated automatically as users surf between sites.

That can't be difficult to do in Domino

… says I. And it isn't. So I've done it. A very simple one mind. It doesn't check back on links, and duplicate filtering is rudimentary to say the least. But it's a start, the code is very "thin" and it does the job. If you want more, check out the link in "Further reading" below. So, how's it done?

The form

First, you need a form to store one or two fields… this is your referrer document, and will be referenced solely by some @DbColumn code. More of that later. Create a computed text field in this form, which computes to itself. Call it whatever. For the purposes of this code, I've called the form "frmReferrer" and the field "txtReferrer". All done.

The view

Now we need a simple view to log these referrer documents. In this example it's called "vwReferrers," and contains two columns. The first, sorted in descending order, contains the code @Created, whilst the second contains a reference to the txtReferrer field thus:

numVal := 23;
numLen := @Length(txtReferrer);
strText := @If(numLen > numVal; @Left(txtReferrer; numVal) + "…"; txtReferrer);
strSite := @RightBack(strText; "://");

"<a href=\"" + txtReferrer + "\">" + strSite + "</a>"

Now, the only value you might want to tweak here is "numVal." This ultimately decides how much of the referrer URL to show in your page within the anchor reference. You don't really want your whole home page taken up by one entry in an innocuous side-panel that rambles on for yonks now do you? I limit mine to 23 chars, and then append "…" — your preference. They key thing is that the full referrer URL is preserved within the anchor reference itself.

OK, so that's the view done. Now the penultimate piece of the puzzle, how to display these referrers in your webpage. I use a subform, which is then inserted at the appropriate point in my web page. Again, you may differ in your approach, but the code will be the same. Create a computed text item, or field if you prefer, with the following code behind it:

numToReturn := 20;
varLookup := @Unique(@DbColumn("Notes" : "Cache"; ""; "vwReferrers"; 2));
varValue := @Implode(@Subset(varLookup; numToReturn); "~");
@ReplaceSubstring(varValue; "~"; "<br/>") + "<br/>"

This looks up to the second column of the referrers view, and returns the requisite number of documents — you can vary the maximum number returned by altering the "numToReturn" variable.

Generating referrer documents

This is all very well, but how does one actually create these referrer documents? Well, what's required is a WebQueryOpen agent in all the forms you wish to generate referrers from. Before eyes start rolling, please note that his shouldn't be cause for alarm unless yours is a really heavily used site. If it is, suck it and see. You may need to go down another route (see further reading below). The code for this agent, like the rest of the stuff here, is very simple. It's some basic Lotusscript which takes the HTTP_Referer (note spelling. Not British!) field from your underlying Domino form / page, and creates the referrer information from that.

Yes, it relies on you having a field called "HTTP_Referer" in your web forms. But you probably do that anyway, as part of your common fields subform whenerver you create a Domino database. If not, you should!

Anyway, assuming this field is present in the relevant forms, your agent code will look like this:

' Specify domains to ignore
Const K_STRING_1 = "yourdomain.com"
Const K_STRING_2 = "localhost"

Dim session As New NotesSession
Dim db As NotesDatabase
Set db = session.CurrentDatabase

Dim doc As NotesDocument
Dim sRef As String

Set doc = session.DocumentContext
sRef = doc.HTTP_Referer(0)
If Len(sRef) = 0 Then Exit Sub

Dim docRef As NotesDocument
If (Instr(1, sRef, K_STRING_1, 1) <> 0 Or Null) Or_
(Instr(1, strRefer, K_STRING_2, 1) <> 0 Or Null) Then
  Exit Sub
Else
  Set docRef = db.CreateDocument
  Call docRef.ReplaceItemValue("Form", "Referrer")
  Call docRef.ReplaceItemValue("refValue", sRefer)
  Call docRef.Save(False, False, True)
End If

See? Simple! OK, so this will only work where you're using forms, but in Domino development, that's pretty often: $$ViewTemplates, documents and such.

Further reading

Comments

  1. Might be a wee bugglet in here, mate…

    Are you sure the line "If (Instr(1, strRefer, K_STRING_1, 1) <> 0 Or Null) Or
    (Instr(1, strRefer, K_STRING_2, 1) <> 0 Or Null) Then…"
    is doing what you want it to (particularly the "<> 0 Or Null" bit)?

    For an example of what I'm on about, run the folloing:

    Dim i As Integer
    i = 2
    If i = 0 Or -1 Then
    Msgbox "Yes!"
    Else
    Msgbox "No!"
    End If

    Using a logic operator after a comparison operator doesn't work the way I think you want it to in your code.

    In my code you'd need to say
    "If i = 0 Or i = -1 Then…" to get it to behave as I think you want it to.

    Have I misunderstood your intentions?

    Cheers. C! 8) Chris Molloy#
  2. Note to self: don't put mark-up in comments. ;-) Cheers. C! 8)Chris Molloy#
  3. I know what you mean mate… good spot! For example, @formula expressed like that definitely wouldn't work. But hey, I'm a cheap hacker. It haven't noticed it not doing what it should, which is to prevent the creation of referrer docs for any links sourced from "benpoole.com" or "localhost" pages. So that's odd. Hmmm…

    BTW, sorry about your pass-thru… got to combat those script kiddies divvent ye knaar!

    Ben Poole#
  4. I've been playing with your referrers technique for a few days now (have not bought out into my UI as yet however) - It has got me thinking of other ways to approach this…

    Being dominodeveloper.net customers, perhaps another technique would be populate a view by running a scheduled agent over domlog.nsf? - that way you're not slowing down a visitors experience.

    I guess this extra effort (much more coding) may be wasted unless there is a significant speed payoff for busy sites.

    Anyway, nice technique Ben - thanks!

    Colin Williams#
  5. Colin, interesting idea.

    It comes down to logistics though… dominodeveloper.net have several development servers (what we're on), plus their production set-up. domlog.nsf can get very big, very quickly. For sites like ours it's probably not worth the pay-off. But agreed, for heavy use sites, you'd probably leverage Domino technology to have a "domlog.nsf of your very own" eh!

    To be honest, whilst an agent is employed, the technique is very lightweight — I've experienced no issues on this site as yet, and so far this month [in cocky git mode now] I've had in excess of 29,000 hits, so so far, so good.

    Ben Poole#
  6. I use a similar agent to store my referer documents. But I call the agent differently. Not in the WebQueryOpen but using a trick I noticed on Notestips.com. I call the agent by including the following computed for display formula in the page/form or any other design element.

    sHTTP := @If( HTTPS = "ON"; "s";"");
    sReferer := @LowerCase(HTTP_Referer);
    @If((sReferer = "") | @Matches(sReferer; "http" +
    sHTTP + "://*swapcode.com*")
    | @Matches(sReferer; "*127.0.0.1*") |
    @Matches(sReferer; "*localhost*");@Return("");"");
    @NewLine + " " + @NewLine +
    " "\" style=\"border:0px;width:1px;height:1px;\"
    alt=\"\" />" + @NewLine + " " +
    @NewLine

    If the referer is empty then the agent is never called. As the html produced by the formula is never included in the page. You do need to pass the referer via the Query_String and parse it in you agent though. As when the agent is called HTTP_Referer is empty. I also store all the CGI variables in my agent.

    John Marshall#
  7. Ben, never trust anything that stems from a visitor. Referrers included! I've set up an example exploit on my website (link above) - it's in german but, well, the HTML should be obvious for everyone.

    wflamme - Home

    BTW, I'd recommend to disallow any HTML in forms, eg: {naughty script tag removed} alert("Oops - this could have been malicios code as well…") { naughty script tag end}. Never mind, keep on with the good work!

    Wolfgang Flamme#
  8. Wolfgang, you can't use script tags on this site — they don't parse and the code removes them on submission: look at what happened to your post! In any case, I can tweak what I want from the Notes back end…
    ;-)

    As for not trusting referrers… of course these can be spoofed; I gather referrer spamming is the latest thing. But I can still edit them from the Notes client, and the server removes each day's referrers anyway.

    Ben Poole#
  9. Ben, I knew you'd parse tags because in the comment form you told I were free to use them :) Just found out referrers are most often not parsed since they are not considered user input… So I think one better takes care of the issue. Wolfgang Flamme#
  10. Just found out referrers are most often not parsed since they are not considered user input… So I think one better takes care of the issue.

    Quite right, referrers often don't get checked out, and they certainly aren't checked in the simple code I posted.

    Now, I could write some Java that nips off and validates the referrer, but what does it actually mean if a referrer validates? It means that there's a website out there with a link to my site within its pages. Not really much use.

    1. This can still be spoofed.
    2. That site could still be "dodgy_pornographers.com"

    So, what else is open to me… well, I don't allow <script> tags anywhere on this site — including referrers. All referrer records get wiped out daily, and I can delete individual records at any time I want.

    For me then, the "risks" — such as they are — outweigh the benefits. It's a call individual coders have to make.

    Ben Poole#

Comments on this post are now closed.

About

I’m a software architect / developer / general IT wrangler specialising in web, mobile web and middleware using things like node.js, Java, C#, PHP, HTML5 and more.

Best described as a simpleton, but kindly. You can read more here.

";