Since our launch one year ago, Crashlytics has set the bar for the most informative crash reports on mobile. Above and beyond stack traces, RAM usage, and disk utilization, we’ve sought to provide all the critical data-points that developers need to pinpoint and fix issues – device orientation, battery state, even whether the device was being held up to the ear! And we’re never satisfied.

A treasure-trove of data lies in an app’s logs and there’s no better way to debug a problem than by knowing exactly what happened leading up to the critical moment. Capturing logging data has been our number-one customer request for months and our number-one concern. We care deeply about security and end-user privacy: collecting logging data opens the door to substantial risks. We wanted to begin the path down the road to building a Splunk for Mobile.

I’m excited to announce that after focusing our R&D efforts, we think we’ve cracked it, and I wanted to share some details on our approach.

Privacy, Performance

The easiest way to deliver logging would be to capture and redirect all output from NSLog(), but this is also the easiest way to infringe user privacy. Many apps don’t take the care they should in scrubbing log lines of personally-identifiable information: names, email addresses, even passwords often appear in URLs or internal settings that might commonly get logged. Sending this data, even encrypted over SSL, would be dangerous and in-breach of most privacy policies.

Instead, we’ve chosen to introduce completely distinct logging functionality called CLSLog(), so it’s explicit what data will be collected and transmitted with Crashlytics reports.

We also took the opportunity to make some performance improvements – in our benchmarks, CLSLog() is 10X faster than NSLog() under the same conditions. Using CLSLog() could not be easier – it’s a drop-in replacement:

1 OBJC_EXTERN void CLSLog(NSString *format, ...); // Log messages to be sent with crash reports

1 NSLog(@"Detected Higgs Boson with mass %f!!", [boson mass]);
2 CLSLog(@"Detected Higgs Boson with mass %f!!", [boson mass]);

Options, Options, Options

Of course, in many case you might want your log messages to also output to the system log, or show up in Xcode’s console. For these cases, we’ve also provided CLSNSLog(), which records the output and then passes it along to NSLog():

1 OBJC_EXTERN void CLSNSLog(NSString *format, ...); // Log messages to be sent with crash reports as well as to NSLog()

But what if both could happen? In development builds, it would be ideal for everything to pass-thru to Xcode’s console so debugging was as easy as possible. In release builds, though, that’s nothing but overhead — it would be great to take advantage of the blinding speed of our 100% in-memory implementation of CLSLog().

We’ve got you covered:

 1 /**
 2 *
 3 * The CLS_LOG macro provides as easy way to gather more information in your log messages that are
 4 * sent with your crash data. CLS_LOG prepends your custom log message with the function name and
 5 * line number where the macro was used. If your app was built with the DEBUG preprocessor macro
 6 * defined CLS_LOG uses the CLSNSLog function which forwards your log message to NSLog and CLSLog.
 7 * If the DEBUG preprocessor macro is not defined CLS_LOG uses CLSLog only, for a ~10X speed-up.
 8 *
 9 * Example output:
10 * -[AppDelegate login:] line 134 $ login start
11 *
12 **/
13 #ifdef DEBUG
14 #define CLS_LOG(__FORMAT__, ...) CLSNSLog((@"%s line %d $ " __FORMAT__), __PRETTY_FUNCTION__, __LINE__, ##__VA_ARGS__)
15 #else
16 #define CLS_LOG(__FORMAT__, ...) CLSLog((@"%s line %d $ " __FORMAT__), __PRETTY_FUNCTION__, __LINE__, ##__VA_ARGS__)
17 #endif

In Debug builds, CLS_LOG() will pass-thru to NSLog, but in Release builds, it will be as fast as possible:

1 CLS_LOG(@"Higgs-Boson detected! Bailing out... %@", attributesDict);

Network Efficient

We’ve designed our custom logging functionality from the ground-up to respect your end-users network connections and your app’s performance. Since it’s implementation is entirely in-process, it’s blazingly fast with no IPC overhead. It also accepts as much data as you choose to throw at it: CLSLog() maintains an auto-scrolling 64kb buffer of your log data, which is more than enough to record what happened in the moments leading up to a crash without exploding your app’s memory requirements or your end-users cellular data plan. Believe it or not, it’s even more memory-efficient than it sounds – our advanced architecture doesn’t even require holding all 64kb in RAM!

That’s Not All…

Viewing logging information is a whole other story. Rather than explain it, I’d encourage you to head over to our SDK Overview and see for yourself! We’re hard at work on additional SDK functionality and have much more to talk about in the coming weeks – stay tuned!

TL;DR: 31 lines of Rack middleware leverage Redis for highly-performant and flexible response caching.

As Crashlytics has scaled, we’ve always been on the lookout for ways to drastically reduce the load on our systems. We recently brought production Redis servers online for some basic analytics tracking and we’ve been extremely pleased with their performance and stability. This weekend, it was time to give them something a bit more load-intensive to chew on.

The vast majority – roughly 90% – of inbound traffic to our servers is destined for the same place. Our client-side SDK, embedded in apps on hundreds of millions of devices worldwide, periodically loads configuration settings that power many of our advanced features. These settings vary by app and app version, but are otherwise identical across devices – a prime candidate for caching.

There are countless built-in and third-party techniques for Rails caching, but we sought something simple that could leverage the infrastructure we already had. Wouldn’t it be great if we could specify a cache duration in any Rails action and it would “just work”?

1 cache_response_for 10.minutes

Rack Middleware to the Rescue

One of the most powerful features of Rack-based Rails is middleware – functionality you can inject into the request processing logic to adjust how it is handled. This will let us check Redis for a cached response or fall-through to the standard Rails action.

 1 class RackRedisCache
 2   def initialize(rails)
 3     @rails = rails
 4   end
 5
 6   def call(env)
 7     cache_key = "rack::redis-cache::#{env['ORIGINAL_FULLPATH']}"
 8
 9     data = REDIS.hgetall(cache_key)
10     if data['status'] && data['body']
11       Rails.logger.info "Completed #{data['status'].to_i} from Redis cache"
12       [data['status'].to_i, JSON.parse(data['headers']), [data['body']]]
13     else
14       @rails.call(env).tap do |response|
15         response_status, response_headers, response_body = *response
16         response_cache_duration = response_headers.delete('Rack-Cache-Response-For').to_i
17
18         if response_cache_duration > 0
19           REDIS.hmset(cache_key,
20             'status', response_status,
21             'headers', response_headers.to_json,
22             'body', response_body.body
23           )
24
25           REDIS.expire(cache_key, response_cache_duration)
26           Rails.logger.info "Cached response to Redis for #{response_cache_duration} seconds."
27         end
28       end
29     end
30   end
31 end

A response in Rails consists of 3 components – the HTTP status, HTTP headers, and of course, the response body. For clarity, we store these under separate keys within a Hash in Redis, JSON-encoding the headers to convert them into a string.

If the cache key is not present, the middleware falls-through to calling the action, and then checking an internal header value to determine whether the action desires its response be cached. The final critical line leverages Redis’ key expiration functionality to ensure the cache is only valid for a given amount of time. It couldn’t get much simpler.

Implementing our DSL

To tie it all together, the ApplicationController needs a simple implementation of cache_response_for that sets the header appropriately:

1 def cache_response_for(duration)
2   headers['Rack-Cache-Response-For'] = duration
3 end

Boom. It was really that easy.

Impact?

This implementation took us only about an hour to develop and deploy, and the effects were immediate. Only 4% of these requests now fall-through to Rails, CPU usage on our API servers has plummeted, and total queries to our MongoDB cluster are down 78%. An hour well-spent. Our Redis cluster also doesn’t sweat its increased responsibility: its CPU usage is up just marginally!

Join Us!

Interested in working on these and other high-scale challenges?  We’re hiring!  Give us a shout at jobs@crashlytics.com. You can stay up to date with all our progress on Twitter, and Facebook.

We had a blast at last night’s Backbone.js MeetUp – it’s great to see such a thriving community here in Boston and to share a few of the insights we’ve had here at Crashlytics about building scalable applications with Backbone.js. The slides from our talk are up on SlideShare for viewing.

We’re looking forward to the next MeetUp and continuing to work with the Boston Backbone community – if you have any feedback or want to get in touch, leave a comment below!

Check out the slides here.