Serving Different Robots.txt Using Rack

While doing an SEO audit for the daily deal API I’m working on, the subject of the robots.txt came up. In addition to our production environment (what you and everyone else see), we also use an “edge” environment. It’s a place we can push the latest and greatest changes and test them before they go live. Edge is an exact copy of production just running on a different domain. Since we didn’t want to get dinged with content duplication we had to disallow spiders from crawling our edge environment. Here’s how we serve different robots.txt files based on environment using Rack within Rails.

  1. Move public/robots.txt to config/robots.txt. This is now your production robots.txt. Any other environment will disallow everything for all user-agents.
  2. Create a RobotsGenerator in lib
  3. Point /robots.txt to the generator in your routes

lib/robots_generator.rb:

class RobotsGenerator
  # Use the config/robots.txt in production.
  # Disallow everything for all other environments.
  def self.call(env)
    body = if Rails.env.production?
      File.read Rails.root.join('config', 'robots.txt')
    else
      "User-agent: *\nDisallow: /"
    end

    # Heroku can cache content for free using Varnish
    headers = {
      'Content-Type'  => 'text/plain',
      'Cache-Control' => "public, max-age=#{1.year.seconds.to_i}"
    }

    [200, headers, [body]]
  rescue Errno::ENOENT
    headers = { 'Content-Type' => 'text/plain' }
    body    = '# A robots.txt is not configured'

    [404, headers, [body]]
  end
end

config/routes.rb

require 'robots_generator'

YourApplication::Application.routes.draw do
  # ...

  match '/robots.txt' => RobotsGenerator

  # ...
end

Update (Sept. 23, 2013): Thanks to @michaelbaudino for pointing out that routes.rb needs the require 'robots_generator' since Rails 3 does not autoload files in lib. Additionally, the request headers should always include Content-Type to avoid a Rack::Lint::LintError error.

Thanks for reading! I'm Avand.

I am a full-stack software engineer, product designer, and teacher. I’ve been working on the web for over a decade and am passionate about building great products.

I currently work at Airbnb, where I work on internal products that help the company build quality software that customers love. Before that, I was at Mystery Science, transforming how elementary school teachers teach science. For a while, I also taught with General Assembly, teaching aspiring developers the basics of front-end web development.

I was born in Boston, grew up in Salt Lake City, and spent many years living in Chicago. Now, I call San Francisco my home.

I enjoy the great outdoors and absolutely love music and dance. Cars have been an lifelong obsession of mine, especially vintage BMWs and Volkswagens. I’m the very proud owner of a van, duh, and he and I plan to one day visit all the national parks and baseball stadiums. Then, off to Central America. A man can dream :)

What can I do for you?

Read my other posts or get in touch: