Macintosh Software Testing Resources

mactester.com: information about testing Macintosh software on the UNIX-based OS X (that's "ten", not "ex"). Intended audience: new and experienced Mac software testers, developers, tech writers and other development project team members, those considering using software testing services. Be a champion for a great user experience. Develop, test and document great Mac apps.

Tidbits

• Best wishes for a productive and bug-free 2012. Have you updated the copyright year (About box, Finder info window, splash screen, documentation, website) for the new year? If you have an automated build system - strongly recommended, even if you're a team of one - you could tweak it to do this automagically in the future.

General Mac information

• EverythingMac Introduction to Mac culture for Mac and non-Mac folks alike
• MacSurfer's Headline News Links to news and current events

Information for Mac developers

• MacTech magazine; once a developer magazine, now more geared toward power users
• O'Reilly MacDevCenter
• Apple developer mailing lists
• Mac (and iOS) testing forum on qaforums.com

Developer events

• Apple Worldwide Developer Conference (WWDC), 2012 date not yet known, probably at Moscone in San Francisco

Finding Mac developers/jobs

• MacTalent
• Runtime Revolution consultants
• Find a REALbasic consultant refrral service (form)

Reference

• Apple Developer Connection (ADC)
• Apple Developer Connection Reference Library: A comprehensive collection of Apple technical resources, including Documentation, Technical Notes, Sample Code, Technical Q&As, and Release Notes.
• Porting to Mac OS X from Windows Win32 API
• TN 1103: Uniquely Identifying a Mac OS Computer
• Developer - Software Licensing
• Mac OS X and 9 invisible files and folders
• Peter Hosey's Mac app checklist: several years old but still good advice, e.g. make sure various plist strings (bundle ID, version, short version, and Get Info) are correct, use a bug database, use version control, have at least one private beta release, keep a detailed version history, check for leaks, have a real website
• Quality Assurance Tips
• Wondering what systems Mac are being used by power users, perhaps to help you figure out which to support? Check the Omni Software Update Statistics page: OS version, preferred language, installed RAM, CPU/speed/count, and other useful system info about Omni users' systems
• Unsanity Smart Crash Reports: allows developers to add crash report submission capabilities to their apps so that they - and not just Apple - can get their app crash reports. Some of the 400+ apps using it include: Sandvox (Karelia Software), TextMate, Cyberduck, Garage Sale (iwascoding), Montage (Mariner Software), Opera, and Audio Hijack (Rogue Amoeba).
• Need a powerful and free text editor? Check out TextWrangler, "little brother" to the highly-regarded and even more powerful BBEdit.

OS X-specific

• Aqua Human Interface Guidelines (last updated 2009-08-20); link available for 394-page PDF
• Mac OS X Technology Overview; also available as PDF

Tools/utilities

• Apple's System Profiler - comes installed on all systems. Tip: system_profiler is the command line version
• InfoPlistChecker checks app's Info.plist (successor to the very useful PlistChecker)
• Cong bundle resource checker reports errors, warnings and notes for such issues as useless framework header files (ScanDrop Lite 1.0.9 had 456!), menus with three periods instead of ellipsis characters, missing symbolic links, deprecated Info.plist keys, missing icon file and more
• OSX AppleScript GUI scripting: System Events application with support for controlling an application's GUI via scripts - select menu items, push buttons, enter text into text fields, and more
• Redstone Software: Eggplant platform-independent automated GUI testing tool
• Mactracker, a freeware app (donations appreciated) for Mac OS, iPhone/iPod Touch and Windows that has system information for Apple products - hardware and operating systems: find out CPU, RAM and other useful info. Thanks, Ian!
• AppleSpec online database of Mac hardware specs

Bug databases

(really, it's much better than storing bugs on scraps of paper, in a text file or an email folder; invest a few hundred dollars to improve your bug management workflow)
• Off-the-shelf
  • Seapine Software's TestTrack Pro cross-platform bug database server runs on OS X, as well as Windows, Linux and Solaris. Well-designed software with workflow and state change smarts; view the Extended Defect Workflow document.
  • Access the popular FogBuz bug tracker with a Mac client like Manicwave's Tickets, which runs on 10.6+ and connects to FogBugz 7 and 8 servers. Another FogBugz client, Lithoglyph's Ladybugz was discontinued in early 2010 but is still available, now for free. Also, the BugzKit Objective-C library used for Ladybugz is available on GitHub.
• Brew your own with...
  • FileMaker Pro
• Other
  • Bugzilla, an offshoot of the Mozilla Web browser project. This is a Web-based bug database. Requires much patience and a fair amount of Unix sysadmin knowledge to assemble all the pieces, install and configure the server. The TestTrack folks do make a pretty good case for buying an off-the-shelf product, rather than spending time and effort trying to assemble a whole bunch of free software (and basing a very important part of your software business on it).

Documentation/testing

• Helpify (345K .zip) from Omni Group's Outliner Extras page "a Python script that transforms an OmniOutliner file into an Apple Help Book, complete with an index, internal links, navigation, and other niceties. The result can be dropped into your own Mac application. All current Omni Group products' help books were created with this tool. The included outline acts as documentation and as a starting template."
• From "The Elements of User Interface Design" by Theo Mandel


• All textual aspects of the interface should be designed by those with writing skills. All writing is an art, including writing system and program documentation and messages... establish the proper tone of voice in messages and prompts. It is important to assign no blame for errors or problems...
• Use terms that users can understand, rather than system or developer terms
• From Aqua Human Interface Guidelines: Has all user-visible text been reviewed by a professional writer?
• From the excellent "Testing Computer Software" (Second Edition) by Cem Kaner, Jack Falk, Hung Quoc Nguyen: How Testing Documentation Contributes to Software Reliability
  • You'll find more bugs than you expect. Surprisingly many bugs show up when a competent tester thoroughly checks the manual (or help) against the program... A full test of the manual takes about an hour per three to five pages...
  • It's an important source of real world combination test cases
  • Bug reports arising out of documentation testing are particularly credible... These are things many people will do. These are errors that magazine reviewers can publish without fear of correction... either the manual will change or the program will change.
  • Use the program exactly as the manual says.
  • Check every statement of fact and every obvious inference from the stated facts, instructions, and suggestions. The manual is the product's specification, and the customer's final place to check whether the program is working correctly.
• A style guide can be used to document proper (and improper) usage of terms - spelling, capitalization, trademarks, copyrights, voice, tense, and so on - to help ensure consistency in all company materials (software, documentation, Web, catalogs, brochures, etc.). Even if you don't have a technical writing department, or even a tech writer, consider starting your own style guide. Apple Publications Style Guide Dec. 2009 (224 pp., 1.2 MB PDF) is an excellent reference.

Creating an installer

Tools

• Apple Installer
• FileStorm MindVision's OS X installer
• MindVision MindVISE Installer; Mindvision offers installer consulting
• Flexera InstallAnywhere multi-platform installation tools

Download/installer suggestions (if you're delivering software outside the Mac App Store)

• Use meaningful filenames for the media, content
• Include a version number so user can plainly see what version will be downloaded; don't make user download and open a file just to find out it's the same version that's already installed!
• Don't install a bunch of files on the desktop. Always enclose files in a folder.
• Give user option to select target folder/disk
• Test for sufficient disk space and tell user how much is required if there's not enough free space

Access to variety of machines, OS versions for testing

• Borrow, barter for services or product; you might try user groups as points of contact
• Apple's ADC Compatibility Labs in Cupertino, California, USA. Mac Developer Program members can purchase a day at the testing facilities at Apple's headquarters (Room With A View, Project X and Field of Dreams) for $100 (previously free for ADC Select and Premier members).
• Local/regional software testing lab facilities
• Rent locally or ship from out-of-town
  • BitbyBit Computer Rentals

Outsourcing your Mac software testing and/or development

• Critical Path Software/eBay (Oregon, USA)
• iBeta (Colorado, USA)
• mactester.com, maintainer of this website (Colorado, USA)
• Recommended Test Labs, Inc. (California, USA)
• VeriTest/Lionbridge
• Sys.Test Labs (Colorado, USA)
• Test Partners Ltd. (UK)

Beta testing

When your application has been thoroughly tested, you may offer a limited beta release to select individuals, and later a general public release. Don't confuse professional software testing with beta testing. There are companies that will help developers set up a beta test program. Refer to Joel Spolsky's Top Twelve Tips for Running a Beta Test (March 2004), which includes these points:
• Plan 8-10 weeks for beta cycle
• New builds at least 2 weeks apart
• Plan at least 4 beta releases
• Most beta testers will try the program when they first get it, then lose interest
• Even with a beta application process, only 20% of testers will actually send feedback

Selling your software on the Internet

• Apple's successful Mac App Store provides developers easy access to the universe of Mac users, and offers a much simpler installation user experience for the customer - no more packages, zip files, disk images, zipped disk images (ugh!); see the Mac App Store Review Guidelines for more info. As noted on various websites/blogs, and thoroughly covered in this Macworld article: The Mac App Store: The devil will be in the details there are numerous restrictions; some of the high-profile points: no demos, no expiring trials, no beta test release distribution, no mechanism for discounts (quantity, education), no paid upgrades. Due to be rejected: crashing apps, apps that don't adhere to proper file system item placement, "deprecated or optionally installed technologies" (Java, Flash, AIR apps), apps with bugs, and violation of terms and conditions in Apple's Human Interface Guidelines.
• FastSpring: Selling through Web Stores vs. Mac App Store (MAS)
• Bodega storefront
• Digital River
• eSellerate
• KAGI (Pronounced "KA" as in COpper, "gi" as in GEEse. Accent on the first syllable. KA-gi.)
• ShareIt
• Yahoo Store

Announcing, distributing your freeware, shareware

• c|net Downloads (formerly VersionTracker) software version updates
• MacUpdate software version updates
• osx.hyperjeff.net: OS X applications
• Tucows shareware reviews and updates
• Association of Shareware Professionals

Getting the word out

• The Cult, user groups. Excerpts from "The Macintosh Way" by former Apple Evangelist and Apple Fellow Guy Kawasaki (Scott, Foresman and Company ISBN 0-673-46175-0)
• Make sure that The Cult uses your product. The Cult is tightly-knit and vocal, and it recommends what it uses. Make sure that The Cult - power users, developers, luminaries, dealers, press, and analysts - use your product by selling it to them near cost, exchanging for their products, or if you have to, giving it away.
• Kill the white space. The Cult wants to make informed purchase decisions by gathering and reviewing in-depth product information. Therefore, your brochures, advertising, and point-of-purchase materials should contain an abundance of technical information, not white space and yuppie lifestyle depictions.
• Promote test driving. If you've got a great product, then a sample package... or a demo kit containing a limited version of your product can significantly increase sales. If you've got a bad product, samples can be good for your competition...
• Don't try to fool user groups
• Never wear a tie. Wearing a tie prevents closeness to a user group. The user group perspective on ties is, "When you ask a Tie a technical question, he has to find a T-shirt to get the answer." (A note to company presidents who are not engineers: take a T-shirt when you visit a user group.)
• Get straight to the demo
• Don't concentrate on only the big groups

Finding Macintosh User Groups

• Apple's User Groups Web site

Software awards (:-)

• Apple Design Award winners
• Macworld Editor's Choice awards
• 19th - February 2004
• 18th - December 2002

Avoid bad publicity (:-(

• Bad reviews - do your best to avoid this sort of publicity:
  • "Program is profoundly flawed... assuming you could get past the bugs, import limitations, and clunky interface. Aspiring desktop publishers on a budget should look elsewhere... barely usable in its current form... Let's hope the next version... fixes the dire shortcomings... Inability to import popular text and graphics formats; amateurish templates; can be unstable..."
  • "Other aspects... are less wonderful. The shipping version was clearly produced in haste; several of the... files wouldn't open, and the awkward dimensions of the shoddy manual make it difficult to use. Instead of a modern HTML-based help system... offers balloons left over from its System 7 incarnation. The HyperCard-based utilities are more nostalgic than impressive in the age of the Web"

Other business advice for developers

• The Do's and Don'ts of Shareware - Sanford Selznick (Sep. 2002)
• Part 1: the foundation upon which to build your emerging software enterprise.
• Part 2: writing and testing maintainable code, assembling a deliverable package, and marketing and distributing your software.
• Part 3: payment processing, distribution, and marketing of your new application. Solid information helpful to beginning and experienced developers alike.
• Media relations and other useful TidBITS
  • Marketing Software, Part 1 by Mike Diegel
  • Hacking the Press by Adam Engst
    • Part 1: Why Bother?
    • Part 2: Types of Publications
    • Part 3: Types of Coverage
  • The Seven Deadly Product Release Sins
  • Successful Shareware by Rick Holzgrafe
    • Part 1
    • Part 2
    • Part 3
    • Part 4
  • Readmes: To Read or Not to Read by Tonya Engst (May 1996)

---


Need help testing your Mac software and documentation? Please email for information about our services. You might also take at a look at this page of considerations for starting a Mac software testing collaboration.


This Web page maintained by: Doug Grinbergs