TL;DR: 31 lines of Rack middleware leverage Redis for highly-performant and flexible response caching.
As Crashlytics has scaled, we’ve always been on the lookout for ways to drastically reduce the load on our systems. We recently brought production Redis servers online for some basic analytics tracking and we’ve been extremely pleased with their performance and stability. This weekend, it was time to give them something a bit more load-intensive to chew on.
The vast majority – roughly 90% – of inbound traffic to our servers is destined for the same place. Our client-side SDK, embedded in apps on hundreds of millions of devices worldwide, periodically loads configuration settings that power many of our advanced features. These settings vary by app and app version, but are otherwise identical across devices – a prime candidate for caching.
There are countless built-in and third-party techniques for Rails caching, but we sought something simple that could leverage the infrastructure we already had. Wouldn’t it be great if we could specify a cache duration in any Rails action and it would “just work”?
1 cache_response_for 10.minutes
Rack Middleware to the Rescue
One of the most powerful features of Rack-based Rails is middleware – functionality you can inject into the request processing logic to adjust how it is handled. This will let us check Redis for a cached response or fall-through to the standard Rails action.
1 class RackRedisCache
2 def initialize(rails)
3 @rails = rails
4 end
5
6 def call(env)
7 cache_key = "rack::redis-cache::#{env['ORIGINAL_FULLPATH']}"
8
9 data = REDIS.hgetall(cache_key)
10 if data['status'] && data['body']
11 Rails.logger.info "Completed #{data['status'].to_i} from Redis cache"
12 [data['status'].to_i, JSON.parse(data['headers']), [data['body']]]
13 else
14 @rails.call(env).tap do |response|
15 response_status, response_headers, response_body = *response
16 response_cache_duration = response_headers.delete('Rack-Cache-Response-For').to_i
17
18 if response_cache_duration > 0
19 REDIS.hmset(cache_key,
20 'status', response_status,
21 'headers', response_headers.to_json,
22 'body', response_body.body
23 )
24
25 REDIS.expire(cache_key, response_cache_duration)
26 Rails.logger.info "Cached response to Redis for #{response_cache_duration} seconds."
27 end
28 end
29 end
30 end
31 end
A response in Rails consists of 3 components – the HTTP status, HTTP headers, and of course, the response body. For clarity, we store these under separate keys within a Hash in Redis, JSON-encoding the headers to convert them into a string.
If the cache key is not present, the middleware falls-through to calling the action, and then checking an internal header value to determine whether the action desires its response be cached. The final critical line leverages Redis’ key expiration functionality to ensure the cache is only valid for a given amount of time. It couldn’t get much simpler.
Implementing our DSL
To tie it all together, the ApplicationController needs a simple implementation of cache_response_for that sets the header appropriately:
1 def cache_response_for(duration)
2 headers['Rack-Cache-Response-For'] = duration
3 end
Boom. It was really that easy.
Impact?
This implementation took us only about an hour to develop and deploy, and the effects were immediate. Only 4% of these requests now fall-through to Rails, CPU usage on our API servers has plummeted, and total queries to our MongoDB cluster are down 78%. An hour well-spent. Our Redis cluster also doesn’t sweat its increased responsibility: its CPU usage is up just marginally!
Join Us!
Interested in working on these and other high-scale challenges? We’re hiring! Give us a shout at jobs@crashlytics.com. You can stay up to date with all our progress on Twitter, and Facebook.
View More
Since starting Crashlytics just over one year ago with my co-founder Wayne Chang, our mission has been clear: build tools that mobile developers love. Now, 14 months later, we’re both humbled and honored to be actively working with many of world’s top mobile apps – Square, Yelp, Groupon, Yammer, PayPal, OpenTable, Waze, HBO, Kayak, Orbitz, Hipmunk, Viddy, Socialcam, and thousands of other organizations.
Today, we are excited to announce that Aaron Levie, the CEO at Box, has joined Crashlytics’ Advisory Board. Levie, named by Inc. as one of the Top 30 Entrepreneurs Under 30, has grown Box into an international powerhouse that now serves 92% of the Fortune 500.
Having raised over $284 million for Box, Levie is building the company for the long-term, to revolutionize how businesses collaborate and share content in the global marketplace.
“The world of mobile has changed dramatically in just the past couple of years,” says Levie, “and with over 5 billion smartphones expected to be in use by 2016, nothing is slowing down. The scale of this market shift radically changes how all applications are built, and requires a new set of tools for the Post-PC development process. Crashlytics sits at the center of this critical part of the ecosystem.”
The state of mobile development is still far from where it needs to be and we look forward to working with Aaron as we hone the next generation of mobile tools.
“Jeff and Wayne are world-class entrepreneurs,” Levie concludes, “and undoubtedly are building an incredibly strong foundation to bring these tools to the masses.”
Follow Aaron Levie at @levie, Wayne Chang at @Wayne, and Jeff Seibert at @jeffseibert.
View More
Backbone.js is a highly extensible framework that provides an MV* paradigm for Javascript applications that we use here at Crashlytics. Its lightweight architecture provides us a great foundation for building modular thick web clients. Backbone’s adaptability allows for the creation of addons and plugins to help tackle many of the problems faced when building a large scale JS application.
However, one of the problems we ran into while using Backbone at Crashlytics was the difficulty of managing the state of our application and its child components in a modular, scalable fashion. Whether it is choosing how to render a view on the DOM or conditionalizing the functionality of a model, the persistent nature of a web application requires a way to manage state-related logic.
Enter Backbone.StateManager
Backbone.StateManager is a module for Backbone.js that adds the ability to manage and utilize states in any size JavaScript application. It can be used as a stand alone object or in conjunction with a target object. Built on top of Underscore.js and Backbone.js, StateManager allows for modular state definitions, sub/pub architecture support with Backbone.Events, transition events between states, and regular expression matching.
Continuing our commitment to open source software, we’re happy to announce that Backbone.StateManager is available today on Github.
Getting Started
The core functionality of StateManager is comprised of 3 simple methods: addState, triggerState, and removeState. Because StateManager extends Backbone.Events, it is also possible to subscribe to the object and listen as it sends out signals when moving between states.
Backbone.StateManager constructor takes two arguments, a state object and an options object, but neither is required. Passed in states will be automatically added and the options are set as an instance property.
1 var stateManager;
2
3 stateManager = new Backbone.StateManager();
4 // or
5 stateManager = new Backbone.StateManager({
6 foo: {
7 enter: function() {
8 return console.log('enter bar');
9 },
10 exit: function() {
11 return console.log('exit foo');
12 }
13 },
14 bar: {
15 enter: function() {
16 return console.log('enter bar');
17 },
18 exit: function() {
19 return console.log('exit bar');
20 }
21 }
22 });
State Definitions
A state is intended to be as modular as possible, so each state definition is expected to contain enter and exit methods that are used when entering or leaving that state. A state definition can also have a transitions property that contains several methods to be used when moving between specified states. There are 4 types of transitions that Backbone.StateManager will defaultly look for: onBeforeExitTo, onExitTo, onBeforeEnterFrom, and onEnterFrom. Each transition is a key value pair, where the value is a method and the key defines the transition type and the specified state (eg onEnterFrom:specifiedState).
1 {
2 enter: function() {
3 return console.log('enter');
4 },
5 exit: function() {
6 return console.log('exit');
7 },
8 transitions: {
9 'onBeforeExitTo:anotherState': function() {},
10 'onExitTo:anotherState': function() {},
11 'onBeforeEnterFrom:anotherState': function() {},
12 'onEnterFrom:anotherState': function() {}
13 }
14 }
Integration
A benefit of StateManager is that it provides an easy method to painlessly add itself to any object. Through the use of the addStateManager, which takes a target object parameter, it reads in any states defined on the target, binds state methods to the target, and creates a new Backbone.StateManager. It also sets a number of convenience methods on the target, including triggerState, getCurrentState, and a reference to the StateManager at target.stateManager.
1 var View;
2
3 View = Backbone.View.extend({
4 states: {
5 foo: {
6 enter: function() {
7 return console.log('enter bar');
8 },
9 exit: function() {
10 return console.log('exit foo');
11 },
12 transitions: {
13 'onExitTo:bar': function() {
14 return 'just exited and bar is about to be entered';
15 }
16 }
17 },
18 bar: {
19 enter: function() {
20 return console.log('enter bar');
21 },
22 exit: function() {
23 return console.log('exit bar');
24 }
25 }
26 },
27 initialize: function() {
28 return Backbone.StateManager.addStateManager(this);
29 }
30 });
Get It
Feedback Welcome
Our goal is to make Backbone.StateManager as useful as possible for you to make writing powerful Backbone.js apps painless. We’re just getting started, and the feedback you provide is critical in achieving this. Please report bugs and discuss features on our Github issues page https://github.com/crashlytics/backbone.statemanager/issues.
Join the Team
Interested in diving-deep into these and other workflow-enhancing challenges? We’re hiring! Give us a shout at jobs@crashlytics.com. You can stay up to date with all our progress on Twitter, Facebook, and Google+.
View More
We’re thrilled to announce Crashlytics Integrations — seamlessly send your Crashlytics issues to your favorite issue tracker!
Crashlytics Intelligence + Workflow you already use = Productivity Win
With Crashlytics Integrations, you get all of the power of our industry-leading crash analysis platform seamlessly integrated into the issue tracking tools you already have, enabling a single-glance view at your outstanding issues. When it’s time to debug, simply jump directly from your issue tracker to the Crashlytics dashboard to get a deep dive into the data.
You’re in control
We care about signal-to-noise ratio — we made sure to build an easy to use control panel so that you’re in full control over when you receive issue updates. By default, we only send along an issue when it reaches Impact Level 2 or higher. However, you can change this to be as verbose as you want.
Launch Integrations
We’re launching Integrations with five of the most-requested offerings. They are:
- JIRA 5
- Pivotal Tracker
- Redmine
- PagerDuty
- Web Hooks
You may be familiar with the first four. The fifth, Web Hooks, is a unique integration that you can use to connect Crashlytics with whatever kind of system you want. We’ll simply POST information to a URL of your choosing, in JSON format. This is great for triggering custom events inside your own development infrastructure or behind your firewall!
Bring-Your-Own
Your options don’t stop there. Faced with the endless array of third-party bug trackers and project management tools available today, it wouldn’t make sense to artificially limit what you can use!
We’re excited to announce that, starting today, we have open-sourced our Integrations implementation so that you can build your own experience and ensure Crashlytics fits seamlessly into your development workflow, whatever it may be. Head on over to GitHub and fork crashlytics-services to get started. We’re looking forward to rolling out the best contributions for everyone to benefit from!
Feedback Welcome
Our goal is to make Crashlytics Integrations as useful as possible for you and leverage the incredible infrastructure we’ve built along with the tools you use everyday to make finding and fixing issues in your apps painless.
We’re just getting started, and we’d love to hear feedback on how we can make this even better for you. Shoot us an email through Support — as you know, we read each and every email that comes through.
Join the Team
Interested in diving-deep into these and other workflow-enhancing challenges? We’re hiring! Give us a shout at jobs@crashlytics.com. You can stay up to date with all our progress on Twitter, Facebook, and Google+.
View More
TL;DR – Objective-C properties are far more complex than they appear. For best results:
- Use nonatomic everywhere
- Make sure you understand the retain-autorelease pattern for getters
- Never, ever use the NS_NONATOMIC_IOSONLY macro
- Always specify setter semantics
- Re-declare readonly properties in class extensions for private setters
- Try out -Weverything, it can be fun
We recently finished up a major update to Crashlytics for Mac, and we’re all really proud of how it turned out. As one of the developers, I always want the code to look as good as the UI. Good style, especially in headers, is something I care about really deeply. So, naturally, when it comes to Objective-C, I use properties.
I’ve been a big fan of ObjC properties since they were introduced in Objective-C 2.0. Prior to that release, getters and setters in Obj-C involved a bunch of boilerplate, but also had some subtlety that tended to easily trip me up. More concise headers and the ability to @synthesize implementations seemed like a mini-revolution at the time. Now it feels weird not to use them, right?
Well, they might seem pretty straightforward, but don’t be fooled: Objective-C properties are complex in both syntax and semantics. In fact, it was my discovery of clang’s “-Weverything” flag at WWDC that made me realize just how complex they can be.
Don’t get me wrong: properties are good, and you should use them. The question is: do you really understand what’s going on?
Declaration Style
Over time, I’ve adopted a very consistent property declaration style. Nearly all of my properties look like this:
1 @property (nonatomic, copy, readonly) NSString* name;
There can be as many as five arguments to an @property statement, and they all have profound effects on semantics. Let’s have a look at them.
Writability
The keywords readonly and readwrite are mutually exclusive, and allow you to control the availability of a setter. readwrite is actually the default, so I’ve always found this keyword to be a little redundant. I’ve also never run across a use case for it, but I suppose you may just not be into the whole brevity thing. There is no way to make a write-only property.
readonly, on the other hand, I use all the time. It’s great for declaring only a getter.
Accessor Method Names
There are ways to control the actual names of the methods that are used for your getter and setter. These are specified with getter=myGetterMethod, and setter=mySetterMethod. I’ve not run into a use case for the setter argument, but as the documention says, getter is useful for adding the “is” in front of boolean properties.
Setter Semantics
Here is where things start getting interesting. Properties allow you to specify five different kinds of semantics for your setter’s behavior.
I use copy for types that contain a single value and might mutate. Consider the following code:
1 NSMutableString* string;
2 string = [NSMutableString stringWithString:@"Dude"];
3 [obj setName:string];
4 [string setString:@"El Duderino"];
Unless you don’t mind your name changing without another call to -setName:, retain is probably the wrong choice!
assign is perfect for value-types, and can be useful for non-owning relationships, like delegates. In these cases, you might prefer weak over assign, because it automagically nils out the pointer should the target object get deallocated.
I personally don’t love the weak reference behavior, because messages to nil fail silently. I could write an entire book on my feelings about Objective-C’s nil-messaging behavior, but let’s sum it up by saying that the liberal use of assertions can help mitigate its side-effects. Given the choice, I’d prefer to crash so at least I know that I’ve made an error somewhere.
ARC also introduced an alternative to retain, which is strong. A little googling shows that there’s quite a bit of confusion about retain vs strong. This isn’t surprising, since they have identical semantics, and produce identical code. Still, strong better communicates the relationship type, so I would recommend using this going forward.
Atomicity
Obj-C accessors sometimes have quite a bit of work to do, and this work is not inherently thread-safe. @synthesize can generate setters and getters atomically on your ivars. This is the aspect of properties that is the most subtle and confusing of them all. I would never recommend using atomic properties and am often frustrated that it is the default. Why?
First of all, the very implication of atomicity sometimes tempts me into thinking I could get away without other kinds of synchronization. How often do you find yourself needing to just read/write an ivar across threads? Sure, it can happen. But, it’s much more likely that the interaction is more complex. If you start off without a lock, others reading your code may not even realize you are depending on an implicit lock. Knowing that all properties are thread-unsafe forces you to consider synchronization more carefully.
Another huge downside is that atomic properties are not recommended on ARM for performance reasons. That the default behavior is inappropriate for Apple’s most dominant platform should be a red flag right away.
Finally, I often find myself writing custom getters and setters. This can be useful for caching an expensive ivar, or just exposing something more complex as a property. Many times I’ve started with an synthesized version, but eventually overridden it. This can lead to a mildly annoying header change, since my custom implementations are typically not atomic. While the compiler seems fine with it, it is a logical error to claim that the default atomic declaration is accurate.
Unfortunately, there’s one additional aspect of atomicity that might be surprising. Let’s call out a section in the Objective-C property documentation:
1 [_internal lock]; // lock using an object-level lock
2 id result = [[value retain] autorelease];
3 [_internal unlock];
4 return result;
If you specify nonatomic, a synthesized accessor for an object property simply returns the value directly.
Yes, this means that nonatomic has a potentially profound effect on the semantics of the synthesized getter that is unrelated to atomicity. There are a host of bugs related to accessors that can be exposed by returning properties without using the retain-autorelease idiom. Take a look at the following:
1 NSString *name = [obj name];
2 [obj setName:nil]; // this could cause name to be deallocated
3
4 // and if it was not retain-autoreleased, this will cause problems
5 NSLog(@"My name is %@, so that's what you call me!", [self name]);
I wouldn’t be surprised at all if this hazard of getters isn’t widely known. There is even a macro in NSObjCRuntime.h called NS_NONATOMIC_IOSONLY that lets you specify the default on Mac OS X but not on iOS. My mind recoils in terror at the suble iOS-only bugs this macro could introduce.
It’s a tough call. Atomic properties have desirable getter behavior. It is just unfortunate that you cannot control getter semantics independently of atomicity, particularly because of the iOS conundrum.
Objective-C documentation, meet Clang
Keeping the documentation in-sync with the vast software systems that is Mac OS X and iOS is a herculean task. I mean no disrespect in this section, but I did find some disconnects that make for an interesting read. I also happen to really like compiler warnings (static analysis of all kinds, really), so I was indulging a little here. I was able to use clang’s -Weverything flag to find a number of places where the compiler and the documentation disagree, sometimes in amusing ways.
1 clang-warnings.m:19:1: warning: property is assumed atomic by default [-Wimplicit-atomic-properties]
2 @synthesize name = _name;
3 ^
4 clang-warnings.m:7:28: note: property declared here
5 @property (copy) NSString* name;
6 ^
7 1 warning generated.
Assumed atomic? According to the docs and numerous stack overflow posts, there isn’t even an atomic keyword. At the time of writing:
“There is no keyword to denote atomic.”
Well, someone tell clang, because this code not only compiles, but also removes the warning:
1 @property (atomic, copy) NSString* name;
Continuing, consider the following code example. This is a very typical use of properties for me: readonly in the header, with a private setter in a class extension.
1 // clang -Weverything -o myclass.o -c clang-warnings.m
2 #import <Foundation/Foundation.h>
3
4 @interface MyClass : NSObject
5
6 @property (nonatomic, copy, readonly) NSString* name;
7
8 @end
9
10 @interface MyClass () {
11 NSString* _name;
12 }
13
14 @property (nonatomic, copy) NSString* name;
15
16 @end
17
18 @implementation MyClass
19
20 @synthesize name = _name;
21
22 @end
Clang doesn’t like this:
1 clang-warnings.m:7:1: warning: property attributes 'readonly' and 'copy' are mutually exclusive [-Wreadonly-setter-attrs]
2 @property (nonatomic, copy, readonly) NSString* name;
3 ^
4 1 warning generated.
There is no mention of readonly and setter semantics being mutually exclsuive. In fact, this exact idiom appears in the docs. But, you could convince yourself that perhaps this warning makes sense. After all, readonly means there is no setter. Typically, I like to include one to provide some additional information to myself and to clients of that property API.
So, what happens if we remove the copy?
1 clang-warnings.m:13:1: warning: property attribute in continuation class does not match the primary class
2 @property (nonatomic, copy) NSString* name;
3 ^
4 clang-warnings.m:5:43: note: property declared here
5 @property (nonatomic, readonly) NSString* name;
6 ^
7 1 warning generated.
This appears to be a bug in clang. I think this is forgivable because -Weverything seems to contain “well-established” warnings that have an associated -W flag, and those that have no such flag, like this one. In this particular case, this warning is definitely bogus. The only way I found to silence it was to remove the setter semantics. This is not acceptable as the setter semantics in combination with @synthesize are critical for correct behavior.
A source of bugs that I love is sending stack-based blocks a retain message. I thought I’d found a great loop-hole in clang’s error-checking with the following:
1 @property (nonatomic, retain) void (^myBlock)(void);
It turns out this is in fact a warning, but only if you have ARC enabled.
1 clang-warnings.m:7:1: warning: retain'ed block property does not copy the block - use copy attribute instead [-Wobjc-noncopy-retain-block-property]
2 @property (nonatomic, retain) void (^myBlock)(void);
3 ^
4 1 warning generated.
While this isn’t in conflict with the documentation, this is a favorite of mine and I just felt that it would really tie the room together.
Summary
While I’m not a fan of the dot-syntax at all (see https://gist.github.com/3150651 for just one of the reasons why), properties are great. Not only can they help to reduce boilerplate, but synthesis can be a significant win for correctness as well. It isn’t great that the compiler and documentation don’t alway agree, but you should definitely be using properties in your own code. Just be sure to understand how they work, because things aren’t always as simple as they appear.
Relevant Radars:
12103527 - Ability to control getter semantics independent of atomicity
12103400 - With -Weverything, clang warns about property redeclaration
12103434 - Clang warnings about mixing writability and setter semantics in properties
12107139 - Objective-C docs claim there is no ‘atomic’ keyword
11921441 - Dot-syntax property does not produce expected error
12103569 - retain block properties should be a warning for non-ARC too
Join the team!
Interested in diving-deep into these and other esoteric challenges? We’re hiring! Give us a shout at jobs@crashlytics.com. You can stay up to date with all our progress on Twitter, Facebook, and Google+.
View More
