How Crashlytics symbolicates 1000 crashes/second

How Crashlytics symbolicates 1000 crashes every second

One of the most complex and involved processes in the Crashlytics crash processing system is symbolication. The needs of our symbolication system have changed dramatically over the years. We now support NDK, and the requirements for correctness on iOS change on a regular basis. As the service has grown, our symbolication system has undergone significant architectural changes to improve performance and correctness. We thought it would be interesting to write something up on how the system works today.

First things first – let’s go over what symbolication actually is. Apple has a good breakdown of the process for their platform, but the general idea is similar for any compiled environment: memory addresses go in, and functions, files, and line numbers come out.

Symbolication is essential for understanding thread stack traces. Without at least filling in function names, it’s impossible to understand what a thread was doing at the time. And without that, meaningful analysis is impossible, whether by a human or an automated system. In fact, Crashlytics’ ability to organize crashes into groups typically relies heavily on function names. This makes symbolication a critical piece of our crash processing system, so let's take a closer look at how we do it.

 

It starts with debug information

Symbolication needs a few key pieces of information to do its work. First, we need an address to some executable code. Next, we need to know which binary that code came from. Finally, we need some way of mapping that address to the symbol names in that binary. This mapping comes from the debug information generated during compilation. On Apple platforms, this information is stored in a dSYM. For Android NDK builds, this info is embedded into the executable itself.

These mappings actually hold much more than needed just for symbolication, presenting some opportunities for optimization. They have everything required for a generalized symbolic debugger to step through and inspect your program, which may be a huge amount of information. On iOS, we have seen dSYMs greater than 1GB in size! This is a real opportunity for optimization, and we take advantage of this in two ways. First, we extract just the mapping info we need into a lightweight, platform-agnostic format. This results in a typical space-saving of 20x when compared to an iOS dSYM. The second optimization has to do with something called symbol mangling.

 

Dealing with mangled symbols

In addition to throwing away data we don't need, we also perform an operation called “demangling” upfront. Many languages, C++ and Swift in particular, encode extra data into symbol names. This makes them significantly harder for humans to read. For instance, the mangled symbol:

_TFC9SwiftTest11AppDelegate10myFunctionfS0_FGSqCSo7NSArray_T_

encodes the information needed by the compiler to describe the following code structure:

SwiftTest.AppDelegate.myFunction (SwiftTest.AppDelegate) -> (__ObjC.NSArray?) -> ()

For both C++ and Swift, we make use of the language's standard library to demangle symbols. While this has worked well for C++, the fast pace of language changes in Swift has proven more challenging to support.

We took an interesting approach to address this. We attempt to dynamically load the same Swift libraries that the developer used to build their code, and then use them to demangle their symbols on their machine before uploading anything to our server. This helps to keep the demangler in sync with the mangling the compiler actually performed. We still have work to do to stay on top of Swift demangling, but once its ABI stabilizes it will hopefully present much less of a problem.

 

Minimizing server-side I/O

At this point, we have lightweight, pre-demangled mapping files. Producing the same files for both iOS and NDK means our backend can work without worrying about a platform’s details or quirks. But, we still have another performance issue to overcome. The typical iOS app loads about 300 binaries during execution. Luckily, we only need the mappings for the active libraries in the threads, around 20 on average. But, even with only 20, and even with our optimized file format, the amount of I/O our backend system needs to do is still incredibly high. We need caching to keep up with the load.

The first level of cache we have in place is pretty straightforward. Each frame in a stack can be thought of as an address-library pair. If you are symbolicating the same address-library pair, the result will always be the same. There are an almost infinite number of these pairs, but in practice, a relatively small number of them dominate the workload. This kind of caching is highly efficient in our system – it has about a 75% hit rate. This means that only 25% of the frames we need to symbolicate actually require us finding a matching mapping and doing a lookup. That's good, but we went even further.

If you take all of the address-library pairs for an entire thread, you can produce a unique signature for the thread itself. If you match on this signature, not only can you cache all the symbolication information for the entire thread, but you can also cache any analysis work done later on. In our case, this cache is about 60% efficient. This is really awesome, because you can potentially save tons of work in many downstream subsystems. This affords us a great deal of flexibility for our stack trace analysis. Because our caching is so efficient, we can experiment with complex, slow implementations that would never be able to keep up with the full stream of crash events.

 

Keeping the symbols flowing

Of course, all of these systems have evolved over time. We started off using hosted Macs and dSYMs directly to symbolicate every frame they saw. After many scaling issues, the introduction of NDK support, and Swift, we've ended up in a pretty different place. Our system now makes use of Twitter's Heron stream-processing system to handle close to 1000 iOS/tvOS/macOS and Android NDK crashes per second. Our custom mapping file solution uses an order of magnitude less bandwidth from developer's machine than it once did. We have a cross-platform symbolication system. On-client demangling produces more correct results than ever, especially as Swift goes through rapid iterations.

We’ve come a long way, and we're always working to improve the process. A few months ago, we released a new tool for uploading debug symbols. Not only does it incorporate all of our dSYM translation optimizations, but is also really easy to use in scripts or automation.

If I had to pick one big take-away from this work, I would say always be willing to consider custom file formats. Translating dSYMs right on the developer’s machine allowed for a dramatic savings in size, much improved privacy of the app’s internal structures, and more correct results. For NDK, the same approach also made for a simpler developer workflow, and cross-platform compatibility. These are very substantial wins for our customers and for our operational costs and maintenance of our systems.

For those of you that would like to hear even more technical details about crash reporting on iOS, you can check out my Flight presentation. Let us know how you enjoyed this look at our symbolication internals, and if there are other areas you might want to learn more about.

We're building a mobile platform to help teams create bold new app experiences. Want to join us? Check out our open positions!

Introducing OOM reporting: a new dimension to app quality

Crashlytics introduces OOM reporting

Stability issues can derail the success of even the best apps – glitchy software repels people. We know that app quality is one of your top priorities and crashes are your worst nightmare, which is why Crashlytics will always alert you when issues arise. We’ll even help you pinpoint their root cause so you can fix issues fast – we’ve got your back!

Today, we’re extending our crash coverage to include out-of-memory (OOM) reporting on iOS. Now, you can see stability from a whole new angle by understanding the impact OOM events have on your app experience.

 

What’s an OOM event and when does it occur?

Unexpected app terminations degrade your app experience and interrupt your user’s session. One type of app termination you’re probably familiar with is crashes, but there’s another unexpected termination that warrants your attention called an OOM event.

An OOM event is an app termination that occurs when a mobile device runs out of memory. All apps need memory to work, but there is only a finite amount available on each device. When an app needs more memory and there isn’t any available, the operating system terminates the app session. To your users, this looks like any other crash, however in reality, this is an OOM event.

 

Our approach to solving a hard problem: intelligent heuristics

OOM events are difficult to report because iOS doesn’t provide any direct mechanism to detect them and they can be caused by factors beyond your app environment and control, such as your app’s memory usage. But because of how important it is to understand your rate of OOM events, we took on this challenge and added OOM reporting to Crashlytics for iOS devices.

Here’s how it works: When you enable the Crashlytics and Answers Kits available on Fabric, we get a stream of live data about your app’s performance. Then, we apply a server-side process of elimination to this data stream to detect OOM events in your app. This detection is based on an intelligent heuristic, inspired by the work of two talented engineers. In other words, we analyze your app’s event stream to come up with an explanation of why it terminated and if we can’t match it up it to a known reason for termination, we count it as an OOM event. And, since no changes to the Answers SDK were required, you will automatically get OOM reporting without needing to do any work (as long as you have both Crashlytics and Answers installed).

This is an example of how your app analytics and stability kits can work together to unlock powerful new insight – something neither kit could do alone.

At Pinterest, we're shipping iOS app updates every two weeks to millions of people. Using the new OOM insights, we've been able to track our memory optimizations and have confidence in the stability of each release.

- Scott Goodson, Head of Core Experience


 

New line of sight into the impact of OOM events on app quality

Our new OOM reporting dashboard will help you understand if OOM events are a problem for your app. Now, you’ll be able to see the overall percentage of app sessions that were unaffected by OOM events across your builds. You can even drill down and see the percentage of OOM-free sessions for individual builds. This will help you answer important questions like, “Is my app being terminated more on one build than another?” and “Is my app seeing more or less OOM-free sessions over the last week?”

OOM-free sessions device breakdown

We’ll also give you a sortable, daily device breakdown so you can compare the raw OOM counts and the percent of OOM-free sessions across different iOS devices. Once you know which iOS device experiences the most OOM events, you can better triage and spend your time investigating problems on your most used devices. After you roll out a fix, you can monitor the number of OOM events on that device to see if your solution helped reduce them. You may even learn that there are some low-end devices that you simply can’t support.

This additional information paints a clearer picture of your app stability across devices and provides clues for troubleshooting.

OOM Device Breakdown

Finally, we’ll show you the total number (i.e., the raw counts) of OOM events across all of your builds versus your top three builds. Use this graph to understand the prevalence and magnitude of OOM events.

OOM Events - Raw Counts

 

Don’t let OOM events crash your app party 🎉

Our mission is to ensure that there are no more sad, unstable apps. By adding OOM reporting into Crashlytics, we’re giving you even more insight into the quality of your app. Don’t be blindsided by OOM issues that disrupt your users’ app experiences (and maybe even cause them to flee!). Monitor your OOM-free sessions, promptly identify when OOM events become a problem, and get valuable direction on where to start your troubleshooting by going to the OOMs page from your Crashlytics dashboard. Check it out and let us know what you think!

Introducing enhanced dSYM tools: stay on top of stability

Missing crashes is frustrating, but not knowing the cause is even worse. App users have low tolerance for buggy apps, which is why stable apps retain more engaged users.

When we launched Crashlytics four years ago, we set out to solve the problem of mobile crash reporting through an easy to use, automated experience. We invested in tools that automated the dSYMs upload process for iOS apps, so you never had to worry about missing a crash.

As the mobile ecosystem evolves, there are complex situations (i.e., Bitcode support or the use of dynamic libraries) that can prevent us from locating the right dSYM to symbolicate your crashes. That's why today, we're excited to unveil our new command line dSYM uploader. Now, you'll never miss a crash report and you'll also have more transparency and the highest quality crash data — even in the most complex situations!

 

Real-time alerts when crashes can’t be processed

It isn’t always obvious when you’re impacted by a symbolication issue. Now, any crash that is missing a dSYM will immediately prompt a banner alert in your Crashlytics dashboard. We’ll even show you the aggregate number of unsymbolicated crashes and the UUIDs of the missing dSYMs, meaning you’ll never waste time guessing which ones to upload.

 

More control over dSYM uploads

dSYMs are essential for crash reporting because they allow us to group crashes into larger issues, isolate the root cause of the error, and provide context around its severity. We continue to invest in flexible, yet powerful tools, that help you automate this process since we know how critical the retrieval and submission of dSYMs is to your stability insight.

That’s why we built upload-symbols, a command line tool, written in Swift, that ships within the Fabric Mac app and our Fabric CocoaPod. It’s incredibly scripting-friendly - give it a try yourself!

 

Automate with fastlane and skip iTunes connect

When you submit iOS apps to the App Store with Bitcode enabled, your app gets recompiled on Apple’s servers. This means Crashlytics doesn’t have access to the debug information it needs to symbolicate your crashes. Luckily, you do, via a dSYM download facility in Xcode.

Instead of manually downloading these files from Xcode and submitting them to Crashlytics, you can use fastlane to automate this tedious process after every release.

Learn more about how to save time by setting up fastlane automation.

Crashlytics_dSYM_uploader_with_fastlane

 

Generous crash reprocessing window for peace of mind

Don’t have time to upload missing dSYMs right away? No problem. We know you’re busy so we’ll hold onto unprocessed crash reports for seven days. Once we receive the missing dSYMs, we’ll process the unsymbolicated crashes. Rest assured, you won’t lose any of the information you need to solve the most critical issues.

 

Highest crash reporting quality

We built our dSYM uploader to stay ahead of the curve. Even though managing debug symbols is getting more complicated, we’ve got you covered with our new Crashlytics updates.

In addition to increased transparency on the number of unsymbolicated crashes and missing dSYMs, we’re now giving you more control over uploading them. And with the power of fastlane, we’ve made the dSYM management as smooth and hassle-free as possible for developers that take advantage of Bitcode. Best of all, our seven-day crash reprocessing window means you don’t need to drop what you’re doing to find a missing dSYM - you have time to retrieve them without risking crash data loss.

Try our new dSYM uploader to get even more insight into your crashes and become the master of your app’s stability. If you have more ideas for how we can make dSYM processing even better, we’re all ears!

Introducing Velocity Alerts

Since the birth of Crashlytics, our passion has always been to help you fix the most prevalent issues quickly and painlessly. Just last month, we released badges to help you hone in on crashes that stand out against the rest.

Today, we’re thrilled to introduce Velocity Alerts: intelligent, proactive alerting on the most critical issues happening in your app. Now, you truly know when you need to take action right away.

 

Powered by the #1 mobile analytics solution

While our system already distills and prioritizes your crashes based on their impact, there was more we could do to make your life even easier. By combining your app’s crash data with usage analytics from Answers, our new Velocity Alerts enable you to triage unexpected, high-impact issues with minimal effort.

Now, once you’ve enabled Answers as part of Crashlytics, our system will proactively check to see if there is a statistically significant number of sessions that have ended due to a crash related to one issue on a particular build. If so, we'll let you know if that issue is a hot patch candidate and needs your attention immediately right on your dashboard. You’ll also get an email alert in your inbox as well as a push notification if you’re using the Fabric mobile app (Android support coming soon). With Crashlytics, you’ll never miss a critical bug!

While at it, we’ve also redesigned all of our notification emails so important information are even more clear and actionable.

We couldn’t be happier to release this so you can sit back, relax and just focus on building the best app. If you’re already using Crashlytics with Answers, Velocity Alerts are automatically enabled for you. If not, get started with Answers!

Crashlytics February Update

We spent February refining our dashboard and thinking about ways we can enhance the Crashlytics experience that many of you have come to know and love. In the last few months, we shipped the ability to log the most critical non-fatal errors on iOS apps as well as a complete visual refresh to the dashboard.

This month, we reexamined the behaviors of alerting and noticed opportunities where we could make your time spent on crashes even more efficient. We're excited to introduce a new way to highlight significant and unique bits of information about particular issues with badges!

The most powerful and intelligent crash reporter

Many crashes often have a common characteristic. For example, a number of issues may have all been related to crashes on a particular device model. With this new feature, we wanted to make triaging your crashes even more efficient on Crashlytics.

With this feature, we highlight when particular pieces of information are significant for an issue and call them out with badges on the dashboard.

After a new release, you'll easily be able to identify if you have any issues that stand out against the rest. For example, you can stop ignoring top issues if they are only prevalent on jailbroken devices, or you can quickly react to problems caused by the latest OS release. With this new feature, you’ll be able to triage many issues like these at a glance.

We were thrilled to hear the positive feedback from many of you!

 

Now on the Fabric mobile app

Last week, the Fabric team launched the new Fabric mobile app and the Crashlytics badging feature is just one of the many insightful things we’ve integrated with the app. Using the Fabric mobile app, you’ll get push notifications about what matters most: full stacktraces, number of affected users, and breakdown of devices and platforms – all in real time. This way, even before you open your laptop, you know where to look for a bug, who to reach out to on your team, and how it could affect your metrics.

We’ll be shipping even more enhancements in the coming months — stay tuned for more!

 

Here’s our internal changelog:

Mac App and iOS SDK

  • Fix: Improved stability when running on OS X 10.7.

  • Fix: Correct a rare case where libz may fail to compress crash reports before submitting. Previously we would drop the report, now we fall back to sending an uncompressed version.

  • Enhancement: Implemented retry logic to handle partial success from the write syscall. This should help address an extremely rare case of report data corruption.

Web

  • Added issues badges to dashboard. 

Integrations

  • Integrated with Fabric mobile app on both iOS and Android. 

Crashlytics January Update

In 2015, Crashlytics was named the top performance SDK and crossed over one million apps integrating our crash reporting SDK. But we didn’t want to stop there. In January, we were heads down enhancing Crashlytics and making it even more instrumental to how you build mobile apps.

Introducing the ability to log NSErrors

Crashes aren’t the only adverse events that can take place in a running app, where errors might occur in the background and affect their behavior or performance. In January, we were thrilled to officially release the ability to log NSErrors on iOS, OS X and tvOSWe wanted to empower you to optimize your user experience even when your app isn’t crashing.

We extended the #1 crash reporting experience so you can easily log and fix the most critical non-fatal errors on Apple devices — all in real time. You'll also notice we gave your issue list dashboard a complete visual refresh. Now, even looking at crashes can be an enjoyable experience :)

Enhancing Unity Support for Crashlytics

Building an amazing game can be one of the most challenging, yet invigorating experiences. We thought we could take the Crashlytics experience and bring it to your Unity projects right out of the box. Since announcing our beta support for Unity at Twitter Flight, we've seen overwhelming demand from many developers who are building some of the most cutting edge games. Starting today, we're opening up access to more beta testers.

Want to get your hands on it early? Sign up at Crashlytics Labs today — we'd love to hear what you think!

Here’s our internal changelog

Mac App and iOS SDK

  • Added OS X support to Answers

  • Gave the issue list dashboard a complete visual refresh

  • Feature: Logging NSErrors is now out of Beta and ready to be used

  • Fix: Correct a regression that can cause DWARF unwinding to fail for x86_64

  • Fix: Correct a race condition that could happen when setting custom keys for the previous crash in the delegate callback method

Introducing the ability to log NSErrors

At last year’s Flight conference, we opened beta access to one of our new tools at Crashlytics Labs: the ability to log NSErrors on iOS, OS X, and tvOS. Beyond actionable insights into errors when an app or game crashes, we wanted to empower you to optimize your user experience even when your app isn’t crashing. Since Flight, we’ve been building this out, while releasing other major features such as Crashlytics for tvOS along the way.

Kicking off the new year strong, we’re excited to share today that we’re officially releasing the ability to log NSErrors on iOS, OS X and tvOS with Crashlytics!

Total Visibility Beyond Crashes

Crashes aren't the only adverse events that can take place in a running app. Errors might occur in the background and affect the behavior or performance while the app is running, leading to poor user experience or other serious problems later. Knowing about these non-fatal errors can help you monitor and address problems before they become more serious, or uncover issues that might be due to a server/back end problem.

With Crashlytics, we’ve extended the #1 crash reporting experience so you can easily log and fix the most critical non-fatal errors on Apple devices — all in real-time. Just like for crashes, you’ll have everything you need: in-depth stack traces, cross version analysis, device/OS breakdowns, among all the other features you already know. 

To get started with the new feature, simply go to your Crashlytics dashboard and select the “Non-Fatals” filter option. Now, all your non-fatal errors will be grouped by their error domain and code, and you'll see the NSLocalizedDescription right there in the list. 

We couldn’t be happier to get this new feature into your hands and look forward to the awesome apps you’ll be building this year. Happy coding!

  

Introducing Crashlytics for tvOS

Last month at Flight, we opened beta access to our crash reporting for tvOS project at Crashlytics Labs. Since then we’ve received much positive feedback from many of you. Today, just a month after Flight, we’re thrilled to share that we’re officially releasing Crashlytics for tvOS!

The #1 Crash Reporting Solution Now on Apple TV

Our passion has always been to deliver the most powerful developer tools. With Crashlytics, we’re solving the biggest challenges in crash reporting for iOS and Android; but we didn’t stop there. Many of you are building apps for TV today, and we’re excited to support the next generation of experience you’re crafting for the living room, starting with Apple TV.

Our tvOS support gives you the same award-winning experience that you’ve come to expect from us, starting with prioritizing your most prevalent crashes down to giving you the exact line of code that caused your Apple TV app to crash. Just like on iOS, there’s no need to wait days for a crash report: we’ll notify you with what’s important in real time so you can fix crashes as they happen. On top of that, you’ll have total visibility into an issue’s life cycle for your Apple TV apps, with a complete picture of how an issue is affecting your builds.

Don’t take our word for it:

Our team is building the next generation of experiences for our users on devices like the Apple TV. The Crashlytics team has done it again. They've brought the same crash reporting tool that we love to Apple TV and it's going to be instrumental for us moving forward.

- Ryan Poolos, CEO


MobilityWare is excited about the upcoming launch of our Solitaire and Blackjack titles on Apple TV. Crashlytics was essential in creating a stable and bug free game. Their SDK quickly integrated into our games taking just minutes to replace the iOS version with the tvOS version. Awesome job!

- Jeffrey Hohenstein, Dir. of Engineering


To get Crashlytics for tvOS, simply download the latest Fabric Mac app which will automatically detect your tvOS build in your workspace. We can't wait for you to try it!

 

Beyond tvOS: Enhanced Notifications for Beta on Android

While the foliage is in full form, we also spent November finding new ways to enhance the Crashlytics experience. We realize how important it is to get timely notifications to your users.

That’s why, on top of tvOS support, we’ve enhanced our notification functionality for your beta release process on Android. Now, your testers will be notified about your latest builds much faster, via push.

From Crashlytics Labs: Unity and tvOS Support, Logged Errors for iOS

Last month, we improved our crash reporting service with enhanced visibility into the life cycle of an issue. Even though Crashlytics is already recognized as the #1 most implemented performance SDK, we knew there was more to do. Many app developers face challenges such as getting deep, actionable crash reports for their games, or gaining visibility into the stability issues that don’t always result in a crash.

Through Crashlytics Labs, we’re always tinkering with new ideas. Today, we’re announcing three new tools in beta: support for Unity, one of the most popular game development engines; support for tvOS; and the ability to log NSErrors on iOS, OS X and tvOS.

The onboarding process for Unity-based projects has traditionally been cumbersome. Our beta support for Unity aims to give you a completely streamlined onboarding experience. Right out of the box, our beta will support both Unity 4 and 5; you’ll no longer have to use your Mac app or Android IDE plugins to start building.

From our friends at Dots, Androidsx and Nordeus:

With Fabric, the idea of one SDK that will not take long to implement but give us a broad reach in terms of capabilities in Unity, is a huge offering.

- Chris Deaner, Director of Engineering,


The onboarding is brilliant, simple and fast.

- Pau Rodriguez, Software Engineer,


Having added a lot of plugins so far, I have to say that this was by far the easiest integration.

- Miša Damjanić, Game Designer,


We also knew that the developer community is excited to build tvOS apps that will live on Apple TV. Because of this, Crashlytics now supports crashes that come from those apps on Apple TVs. Available through Crashlytics Labs, you can now test out our tvOS app support in beta.

Beyond actionable insights into errors when an app or game crashes, we saw another need from many those who want visibility into NSErrors. Just because an app didn’t crash doesn’t necessarily mean that the user had a flawless experience, and our beta support for logging errors aims to help you solve that. Not all serious stability issues are crashes, and knowing about other non-fatal errors can help monitor and address problems before they become more serious, or uncover issues that might be due to a server/back end problem, etc. Starting in beta on iOS, OS X and tvOS, we plan to make NSErrors accessible so you have a way to optimize user experience even when your app isn’t crashing.

If you want to apply to test our new tools and other secret projects we’re working on, sign up here.

Cheers,


Crashlytics #1 in Performance, Answers #1 in Mobile Analytics

Just a few months ago, Crashlytics was named the #1 most implemented performance SDK by SourceDNA, the world's largest database of mobile app intelligence. Not only was it the most implemented, it was more than #2-6 combined! At the same time, Answers, the mobile analytics tool we built just over a year ago, was named #2 on iOS and #3 on Android.

Today, we’re thrilled to share that according to SourceDNA’s newest report, Answers is now the #1 most implemented mobile analytics SDK on iOS — just five months after it was named #2!

Here’s what SourceDNA said about Answers and Crashlytics just a few months ago:

Twitter’s efforts in the mobile ecosystem are clearly paying off. Their tools are some of the most implemented in apps that we’re seeing. It’s a clear sign that they are really making an impact and leading the way with mobile.

-


Since founding Crashlytics, our passion has always been to build developer tools that you love. With both Crashlytics and Answers being #1 in their space, we’re humbled, thrilled and ecstatic with the positive reaction for everything we’ve built. Even though we’ve received this recognition, we don’t plan to be complacent. We look forward to continuing our mission of building the best services for every developer in the world.

 

(If you haven’t yet, see the inside story of how a team of six of us built Answers to be the most popular mobile analytics tool today.)

Crashlytics September Update

Now that fall is here and school’s back in session, we’re studying up on how to make your lives even easier with more enhancements. In September, we shipped a new dSYM upload page and a more detailed user experience for Beta to support iOS 9 and its latest changes.

Seamlessly Drag and Drop dSYMs

This month, we enhanced our settings and user controls on your dashboard. For your iOS apps, there are scenarios that might result in you having a few missing dSYMs, one being if you’ve recently enabled Bitcode. But don’t worry — if your latest build is missing dSYMs, we’ll let you know right away via notification on your dashboard. Simply go to your settings page and drag & drop your dSYMs onto your dashboard. We’ll automatically start processing your crashes against the latest files.

We’ll even show you the most impactful missing ones, prioritizing the list in order for you.

iOS 9 Support for Beta Distribution

Since the release of iOS 9, we’ve been rolling out new updates to support it. iOS 9 changed the install steps for end users, so we updated our device tutorials to show users exactly the required steps are. For your beta testers who are now on iOS 9, we added the functionality to help them automatically download the newly required certificate to enable your beta build on their phones — all right within your tester onboarding flow.

Brand New Integrations

Building great apps takes teamwork and collaboration. This month, we’re excited to release two more integrations to help you further strengthen collaboration. Our first one is with Moxtra, which seamlessly synchronizes everything relevant to your team into a collaborative workspace that can be embedded into any app. With this integration submitted by Raj Sen Sharma, your crash reports and real-time alerts can now be funneled directly into your preferred binder. The second is Chatwork, a collaboration tool that connects individuals and teams of all sizes. With the ChatWork integration, you can now seamlessly send crash reports and other important data as notifications to your team.

Here’s our internal changelog:

Mac App

  • Enhanced settings tab to allow users to be able to upload dSYMs and check to see if the most recent build has a dSYM present

Android SDK

  • Fixed a minor id collection issue that occurred when customers ProGuard their apps

iOS SDK

  • Added Bitcode support

  • Dropped support for iOS 5

  • Enhanced stack unwinding for i386 and x86_64 functions that do not create a stack frame

  • Enhanced support for functions that require evaluating DWARF expressions to unwind correctly

  • Fixed a bug that would prevent DWARF unwinding from working correctly on ARM64

  • An OS X AppKit exception-handling check is now performed even if debugMode is disabled

  • Provided a workaround for a header-related issue with Swift debugging

  • Adopted Objective-C generics, when supported

  • Removed some unnecessary code in our one C++ static initializer

Major Upgrade: Making Issues Even More Actionable

Having deep, actionable data as a developer is the key to building great apps. This is something we’ve kept in mind ever since we embarked on solving crash reporting. The real power of a crash reporting service is to distill mountains of information, organize crashes into groups and prioritize these crashes by impact. In Crashlytics, we already surface the most prevalent issues for you — but we didn’t stop there.

Today, we’re taking crash reporting to the next level by giving you enhanced visibility into an issue’s lifecycle. Now, you’ll be able to better manage your view into issues across versions from the first version that the crash occurred to the most recent version we see it causing a problem. We’ll even tell you if and when the same problem regressed along the way.

Optimal organization of your data

Previously, new issues were scoped to a particular version. Now, your dashboard will automatically merge the data about an issue occurring across versions so you can see that issue in a single row along with the range of versions it affects.                                                                                                    

With this upgrade, and with minimal effort, you’ll have an even deeper understanding of the impact of these issues. You can still filter by a particular version to see how that version is impacted, and when you filter to just one version on the dashboard, we’ll let you know if an issue is also occurring on the other versions.

Total visibility into an issue’s life cycle

Within a crash report, you now have a pristine visual representation of the lifecycle of a specific bug or issue across all versions. You’ll be able to see all of the versions a particular crash is occurring on, which gives you a complete picture of how an issue is affecting your builds. If you need to identify the most affected version, or get a breakdown on the number of crashes by build, your new dashboard has you covered.

By being able to access the impact of bugs and issues across multiple versions, you can fully understand the lifecycle of your app’s bugs or issues. You can see how long an issue has been persistent and its severity to focus on those that matter most, and ship the most important updates.

More powerful: Regression detection & issue ignore functionality

With cross version issue support we also added two new features: regression detection and the ability to ignore issues. Regression detection alerts you when you close an issue, and the same issue appears on a new version later. Don’t worry -- we’ll notify you when an issue creeps back up after you’ve closed it as these regressions are usually warning signs you'll want to pay close attention to.

Sometimes there are things that aren’t regressions, they are just issues you know are going to occur. You may choose not to fix them because they aren’t issues in your code. Now, you can lock and ignore issues that you know will be present for many versions. By having this added control, you can now further fine tune your dashboard to bubble up the true, critical information that needs your immediate action.

Enhanced Answers integration

Besides managing issues, the team at Answers upgraded our integration to allow you to see crash-free users and sessions by versions on your Answers dashboard. Now, you can dive into each build and see the crashes affecting its stability. Learn more about this upgrade in Answers’ August update.

We couldn’t be happier to get these upgrades into your hands and help you to continue to build awesome apps. Log in to check out the new issues dashboard, or sign up if you don’t have Crashlytics yet!


Launching Pinned Builds for Crashlytics

Over the last few months we’ve continued to build out new tools, like NDK support, tester profiles and others, to make your life easier. Since then, we’ve been heads down testing out more tools to make your dashboard even more powerful for you. Today, we’re pumped to release pinned builds, a new feature that makes it easier for you and your team to quickly find the builds that matter to you.

Your most important builds. One click away.

Generally, you have a few builds you care most about. Maybe a few recent ones in production, beta builds and the nightly train. You also have builds that aren’t as important, that might take time to wade through to get to the ones you’re looking for. With our new pinned builds functionality, you can now find the builds you care most about more easily than ever.

Simply “pin” the builds you care most about, and they’ll appear at the top of your dashboard, keeping the most important information front and center. Besides  giving you immediate access, we automatically surface pinned builds to the top of your teammates’ dashboards so it’s even easier to collaborate on fixing crashes. This is especially helpful when you have a large team with hundreds of builds and millions of users.

We have hundreds of builds that we manage on both iOS and Android, so being able to quickly pin and find the ones that matter most is huge for us. Whether they are beta builds, nightly builds or ones in production, being able to manage those better will let us move faster as a team, and make impactful product decisions to benefit our customers much quicker.

- Greg Calvert, Director, Video Product Management


Pinned builds is live today, so you can start using it now on your dashboard. We can’t wait to hear what you think!

We also shipped a bunch of other enhancements in August. Here’s our internal changelog:

Mac App

  • Made enhancements that add support for Xcode 7 beta as well as the ability to automatically find the latest and greatest version of Xcode to help read dSYMs going forward

  • Fixed a couple of bugs to improve the stability of dSYM uploading

iOS SDK

  • Crashlytics Core

    • Added more detail to "isCrash" property comment

    • Improved the accuracy around crash report detection

    • Renamed the property "apiKey" to "APIKey" in Crashlytics.h

Android SDK

  • Crashlytics Core

    • Fixed an issue that would cause logs written during app startup to not be submitted with crash reports

  • Crashlytics NDK

    • Fixed an issue that would cause native keys/logs to fail when called off the main thread

Web

  • Shipped the ability to pin builds on your dashboard

Crashlytics July Update

With summer in full swing, our team turned up the heat this month shipping new features and performance upgrades under the hood that will help you build the best apps even faster.

Making NDK builds run faster

Just a few months ago, we released our official support for Android NDK crash reporting. In addition to bringing the best of our Java-based Android crash reporting to native code, we’ve made enabling NDK support completely seamless with just a few lines of code. But we didn’t stop there. In July, we made our symbol generation process even more efficient which allows us to dramatically reduce the time to build. Now sit back, relax and watch your native crash reports roll in!

Fine-tuning performance and stability

In July, we were heads down making improvements to our iOS SDK, bringing enhanced reporting stability and speed to your iOS apps. First, we made significant improvements to logging performance that increased speeds dramatically. Next, we quickly identified and worked around an early iOS 9 bug and, while we were in there, we made improvements to increase our defensiveness around reading binary image metadata.

We’re working on much bigger upgrades and enhancements, so stay tuned for more in the coming weeks!

Here’s our internal changelog:

Mac App

  • Added CocoaPods support as a way to manage Fabric dependencies

iOS SDK

  • Increased logging speed by improving performance of CLSLog and CLSNSLog

  • Fixed issue with extra LLVM internal symbol in static library causing duplicate symbol errors when linked alongside other code or libraries built with Bitcode enabled

  • Improved resiliency in crash reporting by increasing defensiveness when reading memory during unwind operations

  • Fixed an issue where calling std::terminate directly would result in a crash

Android SDK

  • Made NDK builds run faster by making symbol generation more efficient

Crashlytics June Update

We can’t help much if you missed the summer solstice in June, but now that it’s summer, we do have a slew of upgrades to share. In June, we were heads down shipping new features to Beta to take tester management to a whole new level, along with integrating another popular project management/bug tracking tool.

Visibility into testers: completely streamlined

When we launched Beta by Crashlytics last May, our vision was to build the most seamless beta distribution experience for both developers and testers. Along the way, your continued feedback has been extremely valuable. We’ve added groups support, the ability to share links as the most convenient way to onboard testers to your builds, and crash-to-tester Identification. And in June, we wanted to make tester management even simpler.

Many folks have requested a new feature to provide a comprehensive view into your individual testers and their activity. Based on the overwhelming demand, we’re excited to release tester profiles today!

With tester profiles, you’ll be presented with an intuitive card, which allows you to see your testers’ most recent activity, how long they’ve been a tester and their activity on all builds — centralizing your tester management dashboard. On top of that, managing your testers has never been easier. From revoking access to certain builds to renaming groups and deleting profiles — it’s at your fingertips.

To help seamlessly track down tester activity for a particular build, your tester profiles include all of the builds your testers are active in and other important information, such as how they’re contributing to the feedback cycle of your app.

Instantly locate your latest build

We’ve also heard you need an easier way to manage your builds in Beta. For example, if you know your latest release in the App Store was version 1.2.2, you now have an easy way to find that exact build. Maybe you want to follow up on a particular build where you fixed that tricky bug that was causing major headaches for testers? Now you can easily navigate to find the answer you’re looking for — on both iOS and Android.

Brand new integration

Since you’re constantly multitasking, we know you're relying on Red Bull and project management/bug tracking tools like Zoho Projects to manage and prioritize your crashes.

Our new integration, built by the Zoho Projects engineering team, enables you to automatically generate issues within their service from crashes reported by Crashlytics. Now, you can stay on top of the most prevalent crashes and spend more time on the fun stuff, like coding the next hottest app :)

Here’s our internal changelog:

Crash Reporting

iOS

  • Crashlytics SDK

    • Added a workaround for an iOS 9 bug that prevented us from correctly capturing loaded libraries
    • Improved performance of custom key-value and logging
    • Optimized use of file protection API resulting in improved launch and logging performance
    • Fixed a race condition around launch failure detection that could result in a crash in the CLSFileManager class
    • Fixed a linker bug that could prevent catch or @catch from working correctly
    • Custom keys will now be removed if the value is set to nil
    • Addressed a launch performance regression that could cause the main thread to be blocked on a slow background operation
    • Fixed an issue that would incorrectly default to enabling NSURLSession background uploads in extensions
    • Adjusted the wording of logging messages around background uploads to be more clear
    • Fixed a bug that could prevent one of our memory corruption defenses from being enabled
    • Improved initialization to ensure the singleton sharedInstance is returned by all initialization patterns in Swift   

Android

  • Crashlytics SDK

    • Removed deprecation annotations for static Crashlytics API. Originally we deprecated this in favor of a more testable API, but we heard your feedback that it was still useful, so we've reinstated it as a convenience :)
  • Crashlytics Core

    • Added support for NDK custom keys, user and logs metadata.

  • Crashlytics NDK

    • Fixed bug causing incorrect thread names to be written to NDK threads in rare cases.
    • Added C++ Crashlytics API header.
    • Fixed issue causing NDK crash reports to fail to be processed with a JSON exception in the logs. 

Beta Distribution

iOS

  • Added the ability to search for previous builds

Android

  • Deferred app launch reporting until an Activity is shown on API >=14
  • Added the ability to search for previous builds

Web

  • Shipped new tester profile UI

 

 

Crashlytics May Update

Between the world buzzing about the fight of the century and sipping margaritas on Cinco de Mayo, we were still heads down in May shipping a slew of new features and upgrades. From SDK v3.0 to NDK support, we were excited to share these upgrades with both the iOS and Android communities ahead of Google I/O and WWDC!

Crashlytics SDK 3.0: Taking reliability to a whole new level

Since starting Crashlytics, we've seen billions of crashes, and have learned that it’s impossible for developers to address every crashing scenario users will face. This is why we aim to give you deep, actionable insights into the most prevalent crashes with minimal effort on your part.

But that’s only the beginning: In May, we released SDK 3.0 for iOS and OS XWith SDK 3.0, we've taken all we’ve learned in the past few years and implemented numerous upgrades to make sure we can reliably get crash reports to you -- even in the most precarious usage situations. See how we did it in our full blog post for all the technical nitty-gritty :)

Finally, NDK crash reporting that just works

In May, Crashlytics was recognized as the #1 crash reporting SDK by SourceDNA, the largest mobile app intelligence firm in the world. We were humbled by this recognition, but there’s more to do to make Crashlytics work even better for you. Since we released Crashlytics for Android, we’ve heard that not only do you want deep, actionable reports to fix the most prevalent crashes within the Android NDK; you want something that just works, with no headaches.

That’s why we were excited to release our official support for NDK crash reporting in May. Just as with Crashlytics for Android, it’s great to bring the best of our Java-based Android crash reporting to native code. We’ve made enabling our NDK support completely seamless with just a few lines of code. To protect your app’s secret sauce, our build tools will generate the needed symbols for you – completely client side – before uploading them to our servers.

See the full announcement on how to get started with our NDK support.

 

Here’s our internal changelog:

iOS SDK

  • Improved initialization to ensure the singleton sharedInstance is returned by all initialization patterns in Swift.

  • Addressed a launch performance regression that could cause the main thread to be blocked on a slow background operation.

  • Increased speed dramatically on that background operation.

  • Fixed an issue that would incorrectly default to enabling NSURLSession background uploads in extensions.

  • Adjusted the wording of logging messages around background uploads for maximum clarity.

  • Fixed a bug that could prevent one of our memory corruption defenses from being enabled.

Android SDK

  • Implemented ability to detect and symbolicate crashes in C++

  • Shipped client-side symbol auto-generation functionality

  • Improved retry logic during degraded network connectivity

Android Build Tool Support

  • Implemented NDK support installation and symbols upload functionality on Gradle

  • Implemented NDK support installation with command-line app onboarding and symbol upload functionality on Ant

Web

  • Added support to display native crashes in the crash reporting dashboard

Beta Distribution

  • Shipped the ability to search for builds on the tester side on iOS

  • Made performance and bug fixes to the Beta Air app on Android


 

The Wait is Over: Launching Crashlytics for Android NDK

When we launched Crashlytics for Android two years ago, we took on a challenge: fixing crashes within Android apps. Just recently, Crashlytics was named the top performance tool on Android by SourceDNA, the largest mobile app intelligence firm in the world. But we continued to listen to your feedback on what could be better. Your response: NDK support.

Since then, we’ve been heads down building out our NDK support in Labs. We’ve listened to your feedback and understood that getting insightful crash reports for NDK code is hard. You not only wanted deep, actionable reports to fix the most prevalent native crashes, but also something that would just work without a headache.

Today, we’re excited to announce our official support for NDK crash reporting!

Most powerful, yet lightweight

We’ve brought the best of our Java-based Android crash reporting to native code. We perform a deep analysis of each stack trace to identify the most important frames so you can see the exact line of code that caused the crash and quickly address the issue.

You’ll have access to the same deep information like devices, hardware, versions.  And now, you’ll spend less time finding and more time fixing your native crashes.

First-in-class installation

The last thing you want is a cumbersome setup or complex configuration just to get crash reporting for your NDK code. Before today, NDK crash reporting involved numerous steps: adding external files, jumping through repos, manually provisioning keys, etc. We’ve built our NDK support to be the opposite: it’s seamless and it’s intelligent.

If you’re already using Crashlytics, you won’t have to download any new plugins or build tools. Simply add a few lines of code within your Gradle or Ant-based project, and we'll do the rest. We’ll even pull in all the dependencies you need automatically.

Not building from a standard IDE? No worries. We'll auto-provision your keys and onboard your app from the command line, just as we do with our plugins. Now sit back and focus on the fun stuff, like coding the next killer app!

Don’t worry, your code is safe

Today, many crash reporting tools for NDK still rely on you uploading your unstripped binaries to generate symbol files server-side. This poses significant risk for your code, sending your secret sauce to third parties, not to mention wasting bandwidth by transmitting your full executable.

With Crashlytics NDK support, our build tools generate the needed symbols – completely client side – before uploading them to our servers. You’ll have total control over the upload process, and our symbol upload file is a fraction of the normal size, saving you tons of bandwidth.

Experience the power

Our official support for NDK is already trusted by many top apps in the Android Play Store:

We use Crashlytics daily to improve the quality in our apps: Path, Path Talk and Kong. NDK support by Crashlytics is one of the best tools on the market to find native errors which are not reported by any other tools.

- Dario Marcato,


A critical portion of the Rdio app relies on NDK. Thanks to Crashlytics, we now trust that we have the best tool on the market to monitor it properly.

- Sean Smith,


Integrating NDK Fabric support only took a few minutes and the payoff was huge! Now we can see exactly where we’re having crashes in our native code by function and line number.

- Dallin Dyer,


 

We’ll be rolling out NDK support over the next few days. Once you have access, you'll see a toggle at the top of our instructions page for Gradle and Ant. Simply toggle the button to the right to see how you can enable NDK support within your projects.

 

If you're already using Crashlytics, be sure to upgrade to the latest Crashlytics Kit on Fabric. If you're already on Fabric and you needed NDK support yesterday, give us a shout and we’ll try to get you into to our next batch! 

We can’t wait for you to try it and hear what you think we should tackle next :) 

 

 

 

Just in: Crashlytics #1 in Performance, Answers #2 in Mobile Analytics

When Jeff and I started Crashlytics back in 2011, we were on a mission to help developers build the best apps. Since every app crashes, we started with crash reporting and then built other services like Beta by Crashlytics, our easy and fun beta distribution tool, and Answers, our focused, real-time approach to mobile analytics.

Four years later, Crashlytics has seen over 1 million apps integrate its crash reporting SDK, and it was recently named the most implemented performance SDK by SourceDNA. Not only is it the most implemented - we’re more than #2-6 combined! We’re extremely grateful for all your support as we continue to build the best in mobile crash reporting.

 

Not just Crashlytics, Answers too

We launched Answers last summer, and 10 months later we’re proud to announce that SourceDNA has also reported that Answers ranks as #2 on iOS and #3 on Android in the mobile analytics space.

From SourceDNA:

Twitter’s efforts in the mobile ecosystem are clearly paying off. Their tools are some of the most implemented in apps that we’re seeing. It’s a clear sign that they are really making an impact and leading the way with mobile.

-


We’re humbled, thrilled, and ecstatic with the positive reaction for both Crashlytics and Answers. We look forward to continuing our mission of building services that you love.

Kick off WWDC with Crashlytics + Twitter

For three years, we’ve hosted a party during WWDC to celebrate with the mobile developer community. And this year’s no different: we’re bringing back the party you won’t want to miss. Join us at 1015 Folsom to hang out with top iOS developers and other Apple aficionados as we celebrate another year of all of you who are building amazing apps.

Now that Crashlytics is part of Fabric, we need a bigger stage to dance on. That’s why we’re celebrating with a Fabric party; it’s the same great memorable event as last year, just with a new look.

We can’t wait to celebrate with you and hear all about what you’re building!

Want to crash our party?

  • When: June 8, 2015

  • Time: 8:00 - 11:00 p.m.

  • Where: 1015 Folsom, San Francisco

  • Cost: It’s on us!

(act quickly -- tickets are limited!)

To get excited for the big night, check out the highlights from last year!

Announcing SDK 3.0: Taking reliability to a whole new level

In-process crash reporting is a highly complex problem. All kinds of reasonable assumptions about your environment can break down when you are running code after a process has already failed. It's incredibly perilous, but can also make for some pretty fun challenges. Can you be defensive enough to survive failure and corruption – and still produce a useful report? This is what we really obsess about, and our latest release, Crashlytics 3.0 for iOS and OS X, takes crash reporting reliability to a whole new level.

Implementing effective defenses against memory corruption

The most common cause of report failures by far is memory corruption of some kind. With the 2.0 release, we began using the mprotect API to guard our SDK’s data structures. We used read-only pages where we could, and guard pages to help protect the writeable areas. This worked out fantastically, and produced a huge improvement in reliability.

Since then, we’ve continued to analyze the remaining sources of corruption, and with 3.0, we've gone even further, using specialized on-disk structures to further minimize our exposed memory. This means the next time some code stomps all over your memory, we're even more likely to get you an accurate report.

Reporting exceptions in your OS X apps

Exceptions have a long and complex history on OS X. This is particularly evident in the differences between how UIKit and AppKit interact with and approach exceptions. By default, exceptions can be very problematic for an AppKit app. Under many circumstances, they are caught by the framework itself. Not only does this prevent a crash report from being generated, it can also result in serious damage to the running process. This is due to the general exception-unsafety of Cocoa. While we cannot fix those issues, we can integrate more closely with AppKit. Now, we're far more likely to be able to produce useful crash reports for exceptions on OS X.

Background uploads

We’re also excited to announce that we’ve adopted NSURLSession’s background capabilities. Where available, SDK 3.0 makes use of background uploads for report submission. This is a huge reliability win, particularly for apps that tend to have short user session durations. As a side effect, we also benefit from the OS’s ability to efficiently schedule networking operations. This means a report that might have gone over a power-hungry cellular connection can now wait until Wifi is available. This can translate into reduced power requirements for crash reporting.

Ship with more confidence

We know how shipping goes. You're huddled over the Answers dashboard, waiting for that new version to start ramping up. This is the best part of shipping, and also where stability problems spoil all the fun. That is why we try so hard to build reliability into everything we do. We want to give you the ultimate confidence in your app's stability, so you can get back to enjoying all those fancy graphs.

 

To get SDK 3.0, make sure you're on the new Fabric Plugin and upgrade your kit!

If you're not on Fabric, no worries -- get SDK 3.0 by upgrading your account to Fabric


Our SDK 3.0 changelog:

  • Feature: Background support submission with NSURLSession.
  • Feature: General API for recording non-native language exceptions (Lua, Javascript, C#...).
  • Feature: Support for C++ exception metadata capture (type and .what(), where available).
  • Feature: AppKit integration for much-improved NSException support on OS X.
  • Feature: Support for compact unwind and DWARF unwinding (i386, x86-64 and arm64).
  • Feature: Runtime control over submission behavior as well as the ability to add info to crashes before submission.
  • Enhancement: Adoption of NSProcessInfo's Activity APIs for better developer visibility into Crashlytics' internal operations.
  • Enhancement: Greatly reduced runtime memory footprint, for improved robustness against memory corruption.
  • Enhancement: Improved Swift integration with adoption of nullability attributes.
  • Enhancement: Improved diagnostics for detecting existing crash/exception handlers that will interfere with Crashlytics normal operation.
  • Enhancement: Improved defensiveness during stack unwinding.
  • Enhancement: Improved defensiveness around multiple threads crashing concurrently.
  • Enhancement: Improved behavior when submissions fail due to transient network issues, particularly on iOS 7+ and OS X 10.9+.
  • Fix: Addressed a potential reliability issue for reporting of crashes during launch.
  • Fix: Reports are no longer generated for OS X apps that spawn child processes which then crash.
  • Fix: Bug that would result in rare cases of incorrect stack unwinding for the current thread on ARMv7 and ARMv7s devices.
  • Fix: Improves the performance of validating memory reads in the trivial-fail case.
  • Fix: Adds in explicit warnings for setting the Crashlytics delegate post start.