We use Atlassian’s Bamboo to handle all of our code builds and unit testing, as such it looks for JUnit style XML reports to be generated by whatever testing framework is being used. Unfortunately sbt doesn’t appear provide any means to tell our testing framework (in this case ScalaTest) to output xml reports as of yet. Solution? Write an sbt plugin that spits out something resembling JUnit XML reports

You can find the project here https://github.com/bseibel/sbt-simple-junit-xml-reporter-plugin its pretty basic and could use a bit of loving but its enough to allow Bamboo to report on our tests.

To include it in your project add a plugin.scala into your projects projects project directory (ha! that is ./project/project/plugin.scala) and drop in:

import sbt._

object Plugins extends Build {

  lazy val plugins = Project(“plugins”, file(“.”))

    .dependsOn(

      uri(“git://github.com/bseibel/sbt-simple-junit-xml-reporter-plugin.git”)

    )

}

Currently its just hard coded to drop reports into `cwd`/test-reports/ but that can be changed easily enough.

Take for example the following string:

key1=value1; key2=value2; key3=value3

In java (or other imperative languages for that matter) how would you go about writing a method that can retrieve the value for a given key out of that string?

Something like this maybe, pretty verbose.

    String extractValue(String key, String string) {

        Pattern p = new Pattern.compile(“(.*)=(.*)”);
        ArrayList pairs = new ArrayList();

        for (String i : string.split(“;”)) {

            pairs.add(i.trim());

        }
        for (String i : pairs) {

            Matcher m = p.matcher(i);

            if (m.matches()) {

                String k = m.group(0);

                String v = m.group(1);
                if (k.compareTo(key) == 0) {

                    return v;

                }

            }

        }
        return null;
    }

In scala thanks to its incredibly powerful pattern matching capabilities we can convert this into something really nice:

val ValueExtractorRE = “”"(.*)=(.*)”"”.r

@tailrec

final def extractValue(name: String, values: List[String]): Option[String] = {

  values match {

    case ValueExtractorRE(x, y) :: rest if x == name => Some(y)

    case x :: rest =>extractValue(name, rest)

    case _ => None

  }

}
val myVal = extractValue(“key1″,string.split(“;”).toList.map(_.trim)))

To me this is infinitely more elegant and readable and really demonstrates the power of Scalas pattern matching.

The important bit to know here if you’ve never used Scala before is that matching is eager, so the first case to match wins. I’ll break the first case into two parts:

1. The :: operator is extracting the head and remaining entries from the list, a simpler form of this is seen in the second case.
2. We try to match the head entry using the ValueExtractorRE Regex from the top and extract the matched groups into x and y. The case has a guard too so we only match if x equals the value we’re looking for, if it does we return an Option as Some(y).

Whats extra-neat is that the compiler will optimize this into a loop with a goto instead of actually being recursive, and I ensure that by providing the @tailrec annotation. If for whatever reason the compiler can’t tail-optimize the method it will bomb with the reason why, which is also pretty useful.

I don’t know if I’ll be able go back and use any other language again

Cheers.

Something is afoot in the above graph, what I can tell you is at around 10:15pm is when I disabled transparent hugepages on this box. So where why am I suddenly buffering and caching more?

What are transparent hugepages?

From what I understand hugepages were introduced to maximize the use of the TLB (translation lookaside buffer) which is a fixed size CPU cache used to speed up virtual memory address to physical memory address lookups. Hugepages combines multiple smaller pages into one huge page such that the TLB can potentially contain more useful pages as a whole and we can skip deep lookups more often. Now this really only matters for systems that are allocating large amounts of memory and in fairly sizeable chunks, or so I assume.

Where did my vfs cache go?

The reason this attracted my attention was because we were having IO problems on a few of our RHEL6 boxes that normally had no issues when run on identical hardware and running FreeBSD instead. The reason was pretty clear, we’re hardly caching anything and we keep going to disk for frequently accessed blocks, but why?

I bet it has something to do with this statement from https://lwn.net/Articles/423584/:

“The current patch only works with anonymous pages; the work to integrate huge pages with the page cache has not yet been done.”

These boxes are primarily running apache with mmap enabled, I’m guessing that when we mmap a file (which is 99% of the time going to be far bigger than one 4k page) we’re allocating a hugepage, and as such skip any sort of page cache for future accesses of that same data. I could be entirely wrong on how those internals work as I havent looked too deeply but it would certainly explain why we’re seeing this behaviour.

So this begs the question of why this is enabled by default on RHEL6, normally they’re pretty conservative on potentially performance breaking features. I can hazard a guess that it’s because the majority of RedHats customers are running large JVM installations where hugepages would actually help and not small apache setups like we are.

Oh well, for now: echo never > /sys/kernel/mm/redhat_transparent_hugepage

Well I too can play this game; so here is my list of 5 things toxic to scalability.

1. Don’t use Object Relational Mappers

Sure any ORM will be your bottleneck when you need to scale into the millions, but lets be honest you’re not writing an app today to support a million users, likely not even 100k. But throw MySQL’s amazing query planner into the mix and you’ll be lucky to hit 5k! Clearly anyone can see why every ORM is bad for scalability here!

2. Don’t worry about data consistency. Live in the eventually, not in the now.

If theres anything Facebook has shown us, it’s that consistently broken images, confusing randomly changing data you don’t expect is A-OK! Atomic operations are so last century.

3. Make sure you have graphs.

Without graphs, executives have no idea how to respond to scalability issues. I mean shit, the site is offline, but show me the graphs that show me that!

4. Scale your database.

It turns out you can’t scale with only one database serving up requests, so you should have more. Unless of course you use NOSQL or something, that shit scales itself right?

5. Feature Flags

Ensure you can turn off various features of the site, like all the interactive bits that made your site worth visiting in the first place.

So it turns out all is not lost, you can still revert to the original behaviour of apples resolver! They’ve added a parameter to mDNSResponder called
-AlwaysAppendSearchDomains. Implying that this new behaviour was very intentional. I had read that Windows apparently made a similar change in one of there past updates as well so I guess this is to help fight some phishing attacks maybe? Either way, tres-annoying!

Anyway the gist of how to fix it is this:

Open up /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist and add
-AlwaysAppendSearchDomains following parameter to the list in the ProgramArguments block:

Then reload the launchd config for it, this should take care of restarting it as well:

launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist


And…

icarus:~ bseibel$ ping util01.tor
PING util01.tor.verticalscope.com (67.223.104.5): 56 data bytes


YAY!

I make heavy use of search domains in my DNS config, as should any respectably lazy systems administrator, so what do I find when I try to be extra lazy this morning by not typing full domain names? Some weird shit, let me tell you:

host-253:~ bseibel$ host util01.tor
util01.tor.verticalscope.com has address 67.223.104.5
host-253:~ bseibel$ ping util01.tor
ping: cannot resolve util01.tor: Unknown host


Uh, what? But wait theres more.

host-253:~ bseibel$ host web01
web01.verticalscope.com has address 74.86.67.166
host-253:~ bseibel$ ping web01
PING web01.verticalscope.com (74.86.67.166): 56 data bytes


So, the domain searching partially breaks when you type more than just a single part? How does something like this happen? This is terribly broken, shame on Apple.

I’ll do some digging and see if I can’t find a real fix for this.

UPDATE:

Interestingly enough even though both are only linked to /usr/lib/libSystem.B.dylib, host is the only one that actually reads /etc/resolv.conf

open_nocancel("/etc/resolv.conf\0", 0x0, 0x1B6) = 4 0
fstat64(0x4, 0x7FFF6EDCCF38, 0x7FFF6EDCCFFC) = 0 0
read_nocancel(0x4, "#\n# Mac OS X Notice\n#\n# This file is not used by the host name and address resolution\n# or the DNS query routing mechanisms used by most processes on\n# this Mac OS X system.\n#\n# This file is automatically generated.\n#\nsearch office.verticalscope.com vertic", 0x1000) = 305 0


And as the file itself would indicate, host is using the traditional unix resolver and the apple ‘custom’ one is broken. Sad.

Now for all the great safety-net like services this provides us, unfortunately their implementation and support model is a bit out of whack. The only way to make use of this off site storage for your backups is via their appliance, which also must store local copies of the backups to be sent off site, so in reality your off-site backups are somewhat tied to your local backup machine functioning, sure, I can accept this. Enter their hardware/support pricing model:

What you’re paying for.

• A box full of mostly consumer hardware
• Slow’ish Celeron CPU
• 2G of ram
• 4 or more enterprise grade disks

What do you do when you’ve built your entire infrastructure on one vendors ‘cloud’? You’ve tied in all of their API’s into your application to make the most of the proprietary features their cloud provides. But what happens when there’s a catastrophic failure and your business is offline for days. Sure you probably should have spent the extra money to ensure that even the partial outage didn’t take you down completely, but not everyone has the money to be able to do that.

I’m sure theres a few CTO’s and VP’s out there that would have taken this recent outage as a sign to consider your alternatives. But what alternatives do you have when your application is so tightly relying on CloudA’s feature set? Well hopefully Cloud Foundry starts to solve this problem. Sure you won’t be migrating your existing application today, but hopefully the idea of a fully open source standard to cloud deployment with no restrictions on what services your cloud can provide will begin to change the landscape a bit.

What Cloud Foundry brings to the table is the idea that there is an open common interface for creating, managing and deploying your application backed by the open source community to help add support for new languages, services and features. Suppose your business doesn’t let you host your applications in a public cloud, well now you have an open source ‘cloud’ alternative that lets you leverage the simplicity of deployment and scalability that the cloud environment provides while still keeping the hardware infrastructure in house.

Look Ma, My own cloud.

Out of the box there is already support for a lot of common frameworks, services and languages. Spring and Grails for Java apps, Rails and Sinatra for Ruby, Node.js apps even. The basic services including MySQL plus some newer ones like Redis and MongoDB. I’m sure now that the source code is out there we’ll very quickly see support for more.

In a recent blog post they discuss the core of the software called VCAP which is claimed to be completely distributed, redundant and allows for hot-scaling of your ‘cloud’ which is something I can appreciate. I’m only slightly disappointed that they decided to use Ruby as a language to write this in, mostly for personal reasons as I simply detest the language. But the idea intrigues me enough that I might just have to get over it.

Another thing that may need to be sorted out is the ability to get your data out of one of these clouds as it doesn’t seem that there is any quick and dirty way of exporting your MySQL databases or other data stores yet (mind you I havent looked heavily into this). Something like this will be key if you want to have the option of moving your applications in the case of major outages and failing service expectations with another provider. Mind you it can be as simple as writing and deploying another app that exports this for you but that seems like a whole lot of effort for something that should be quite simple to accomplish.

In the end I’m sure we’ll see how successful VMware is with its open platform as a service idea by the number of new cloud hosting providers that start popping up. Interesting times indeed!

Its common to see servers wide open to the internet with little to no concern for the traffic inbound or outbound; SSH available on the default ports and remote root access enabled. Rarely have I seen many people require explicitly allowing access to administrative ports from our IP addresses, requiring public keys or even requiring a unprivileged account and providing sudo access.

I can almost state the exact opposite for security practises overseas; Is caring about even basic security practises too hard for the American hoster to bother with?

Control Panel products that have been around for ages have had support for these things for quite some time, but why aren’t they used or set up to be enabled by default at least. Every time I encounter one that is using one of these control panels the feature is left un-configured; Its kind of scary. If hosting companies overseas can do it, surely the US equivalents can too.

I don’t have too much experience with any Canadian hosting providers but I’m curious how we stand comparatively.

So why Grails you ask? Well a few reasons to start:

• It runs on the JVM, probably the most advanced, stable, and fastest (for a non-natively compiled code) piece of software around.
• Groovy: a fully java language compatible less-verbose JVM language
• Hibernate: The de-facto standard when it comes to an ORM

Together this means some seriously rapid development.

So thats all well and good but the real reason why I’m writing this post is because of the huge number of plugins people have made for the grails framework. If you can think of something you need, its pretty likely someone has written something for you already.

So why use plugins when you can just use the libraries themselves? Well the nice thing about the Spring framework that Grails bases itself upon is that plugins are able to inject functionality into your domain classes automatically, bringing you instant gratification when it comes to library usage.

For example, I recently wanted to allow searching through the gobs of data I’m collecting in an easy and maintainable way. The first thing I do is look through the plugin repository for any plugins that may do this for me and it turns out there is, “searchable”!

Simply by installing this plugin and adding  a single static property to any of my domain classes that I would like to be indexed I get an extremely powerful Compass/Lucene powered search engine with almost no effort. With a little more effort I can create complex mappings that allow searches to return relavent objects associated with the root result set with ease.

Example:

    static searchable = {
        name boost: 2.0
        site component: true
    }

This really exemplifies the Grails doctrine of convention over configuration. That closure in my domain class automatically configures searchable to say that the name property gets a bit more priority in results and that the reference to the site domain model is a component of this one, so I will get results for it as well when this node is matched in a search.

So how do you perform a search? Suppose I want to now search through a particular set of instances of a Domain class, thanks to the method injection I now have a “search” method added to all of my domain classes. I simply:

MyDomain.search("my string")

Wow that was easy, heh.
 

Total time to implement site wide search? 15 minutes.

Thats why.