The joy of programming

Lets build our own xcodebuild!

March 08, 2019

The xcodebuild is the tool used to compile iOS & macOS projects from commandline.

We can write our own. It’s simpler than you think!

Why?

What if you could modify Xcode so it works just the way you want?

🚪 Gain access to Xcode APIs

The Xcode is built from a lot of modular specialized frameworks. Use them in your app any way you want.

  • Modify project files programmaticaly
  • Control compilation precisely
  • Generate custom outputs efficiently
  • Anything Xcode does, you can too, in your own way, in the language you love most

💡 Learn how Xcode works internally

Generate Objective-C pseudo-source-code and framework headers for all of Xcode.

🦋 Debug xcodebuild

Run your custom xcodebuild from within Xcode to debug Apple’s frameworks.

📚 Find undocumented features

If you’re curious, you’ll find tons of useful undocumented features in Xcode.

🚀 Add new features

Modify the behavior of Xcode by subclassing and/or swizzling Apple’s code.

💯 Remain always 100% compatible with Xcode

And best of all, unlike 3rd party frameworks, it always works because it is the same code that Xcode IDE uses.

Reading the xcodebuild source code

Lets figure out how xcodebuild works internally.

Find xcodebuild binary

To find the xcodebuild command that will be run in bash use the which command:

which xcodebuild
/usr/bin/xcodebuild

The /usr/bin/xcodebuild is a helper for when you have multiple versions of Xcode installed. It will launch the xcodebuild version you want, based on your choice set in Xcode Preferences or xcode-select.

It is very similar to using the xcrun command. I will describe how they work internally in upcoming post.

Use the xcrun command to find the xcodebuild binary we want:

xcrun -f xcodebuild
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild

Disassemble the xcodebuild binary

You don’t have to know the assembly language.

I recommend the Hopper Disassembler app. It can convert assembly to pseudo code that looks similar to Objective-C. It can also generate Objective-C header files, so that you can see all the methods & properties of all the objects.

Feel free to skip disassembling and use the snippet below. You can always revisit this later.

PS: If you haven’t heard about Grace Hopper, go read about her now: https://en.wikipedia.org/wiki/Grace_Hopper

hopper.png

The main() method

In the main method, we will find that 80% of the code is just:

#import <Foundation/Foundation.h>
#import "Xcode3CommandLineBuildTool.h"
int main(int argc, const char * argv[]) {
@autoreleasepool {
NSProcessInfo *processInfo = [NSProcessInfo processInfo];
// Initialize the Xcode object that does actual work
Xcode3CommandLineBuildTool *buildTool = [Xcode3CommandLineBuildTool sharedCommandLineBuildTool];
// Set name to be used in output
buildTool.name = [[processInfo.arguments firstObject] lastPathComponent];
// Pass all commandline arguments, except for the path to binary
buildTool.arguments = [processInfo.arguments subarrayWithRange:NSMakeRange(1, processInfo.arguments.count-1)];
// Use the same environment variables as the myxcodebuild binary
buildTool.environment = processInfo.environment;
// Use standard output & error for logging, ignore standard input
buildTool.standardError = [NSFileHandle fileHandleWithStandardError];
buildTool.standardOutput = [NSFileHandle fileHandleWithStandardOutput];
buildTool.standardInput = [NSFileHandle fileHandleWithNullDevice];
// Run the command
[buildTool run];
// Return exit code
return (int)buildTool.exitStatus;
}
}
view raw main.m hosted with ❤ by GitHub

Create a myxcodebuild commandline tool

new-project.png Lets start by creating a new macOS commandline tool called myxcodebuild. Use Objective-C rather than Swift as the language.

Add Xcode framework headers

The Xcode frameworks do not include headers, so lets write or generate our own ones.

  • You don’t have to declare all the objects
  • You don’t have to declare all the methods & properties
  • You can change type of any object property to id
  • You can forward declare any object type with @class Type;
  • You can forward declare any interface type with @protocol Type;

Declare just the parts you want.

Add Xcode3CommandLineBuildTool.h

Lets add a Xcode3CommandLineBuildTool.h header:

#pragma once
#import <Foundation/Foundation.h>
@interface Xcode3CommandLineBuildTool : NSObject
@property long long exitStatus;
@property (copy) NSString * name;
@property (copy) NSArray * arguments;
@property (copy) NSDictionary * environment;
@property (retain) NSFileHandle * standardInput;
@property (retain) NSFileHandle * standardOutput;
@property (retain) NSFileHandle * standardError;
+ (id)sharedCommandLineBuildTool;
- (void)run;
@end

This will let the compiler know that such object exists in one of the frameworks.

Later it’s useful to have access to all the methods and properties of Xcode3CommandLineBuildTool. The Hopper app can generate complete private headers.

Implement the main method

Add the decompiled snippet to the main method in main.m:

#import <Foundation/Foundation.h>
#import "Xcode3CommandLineBuildTool.h"
int main(int argc, const char * argv[]) {
@autoreleasepool {
NSProcessInfo *processInfo = [NSProcessInfo processInfo];
// Initialize the Xcode object that does actual work
Xcode3CommandLineBuildTool *buildTool = [Xcode3CommandLineBuildTool sharedCommandLineBuildTool];
// Set name to be used in output
buildTool.name = [[processInfo.arguments firstObject] lastPathComponent];
// Pass all commandline arguments, except for the path to binary
buildTool.arguments = [processInfo.arguments subarrayWithRange:NSMakeRange(1, processInfo.arguments.count-1)];
// Use the same environment variables as the myxcodebuild binary
buildTool.environment = processInfo.environment;
// Use standard output & error for logging, ignore standard input
buildTool.standardError = [NSFileHandle fileHandleWithStandardError];
buildTool.standardOutput = [NSFileHandle fileHandleWithStandardOutput];
buildTool.standardInput = [NSFileHandle fileHandleWithNullDevice];
// Run the command
[buildTool run];
// Return exit code
return (int)buildTool.exitStatus;
}
}
view raw main.m hosted with ❤ by GitHub

Compile myxcodebuild

To compile succesfully myxcodebuild needs to be linked with all the required frameworks.

Find xcodebuild dependencies

Each macOS binary usually has a list of dependencies inside of it.

One way to read the xcodebuild dependencies is:

otool -l /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild

This will output a list of load commands present in the binary, among them library dependencies and their search paths:

...
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 96
         name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1555.10.0
compatibility version 300.0.0
...
Load command 23
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../../../PlugIns (offset 12)
...

Find libraries used by xcodebuild

The LC_LOAD_DYLIB load commands define the libraries that this binary will load on launch:

name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
name @rpath/DVTFoundation.framework/Versions/A/DVTFoundation
name @rpath/DVTDeviceFoundation.framework/Versions/A/DVTDeviceFoundation
name @rpath/IDEFoundation.framework/Versions/A/IDEFoundation 
name @rpath/Xcode3Core.ideplugin/Contents/MacOS/Xcode3Core
name /usr/lib/libSystem.B.dylib
name /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 

There are 3 Xcode frameworks loaded directly: DVTFoundation, DVTDeviceFoundation, IDEFoundation and a Xcode3Core plugin, which is very similar to a framework.

Find library search paths used by xcodebuild

The LC_RPATH load commands define the search locations (the @rpath) for libraries:

path @executable_path/
path @executable_path/../../../Frameworks
path @executable_path/../../../SharedFrameworks
path @executable_path/../../../PlugIns

When macOS is loading a dependency with @rpath in the LC_LOAD_DYLIB path name, it will try all of those search paths in order.

Those are relative to location of xcodebuild inside Xcode.app bundle, effectively:

/Applications/Xcode.app/Contents/Developer/usr/bin/
/Applications/Xcode.app/Contents/Frameworks/
/Applications/Xcode.app/Contents/SharedFrameworks/
/Applications/Xcode.app/Contents/PlugIns/

Add libraries as dependencies to myxcodebuild

We can use the Xcode libraries as they are inside Xcode.app, we don’t need to copy them.

Tip: The Linked Frameworks And Libraries list on the General tab of myxcodebuild target can add frameworks which aren’t on the list. Use the Add Other... button to locate them.

To navigate inside the Xcode.app bundle press Command+Shift+G in the picker window and paste the path to directory in which framework is located: add-other.png

linked-libraries.png

Add Xcode Plugins

For plugins we need to add the binary directly rather than adding the directory.

Add Xcode3Core binary to Linked Frameworks And Libraries using the Add Other... button:

  • /Applications/Xcode.app/Contents/PlugIns/Xcode3Core.ideplugin/Contents/MacOS/Xcode3Core

Add Xcode Frameworks

Add framework bundles to Linked Frameworks And Libraries using the Add Other... button:

  • /Applications/Xcode.app/Contents/Frameworks/IDEFoundation.framework
  • /Applications/Xcode.app/Contents/SharedFrameworks/DVTDeviceFoundation.framework
  • /Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework

Verify library order

The order in which libraries are listed is important - they should appear on the Linked Frameworks And Libraries list in the order they are listed in otool -l.

You can have them in another order, but this may require adding more search paths.

Add custom build settings

Lets use an xcconfig file to keep new custom build settings in one place.

xcconfig-all.png

Add Framework search paths

For the project to compile, linker needs to know where to find the frameworks:

FRAMEWORK_SEARCH_PATHS = $(inherited) /Applications/Xcode.app/Contents/Frameworks /Applications/Xcode.app/Contents/SharedFrameworks

LIBRARY_SEARCH_PATHS = $(inherited) /Applications/Xcode.app/Contents/PlugIns

Add @rpath search paths

The frameworks will be loaded from specific locations when myxcodebuild is launched:

LD_RUNPATH_SEARCH_PATHS = @executable_path /Applications/Xcode.app/Contents/Frameworks /Applications/Xcode.app/Contents/SharedFrameworks /Applications/Xcode.app/Contents/PlugIns

Launching myxcodebuild

Our myxcodebuild is now ready!

As bonus, lets launch and debug it from within Xcode.

We need a project on which it can be tested, feel free to use your own or create a sample app.

Create a sample app to be compiled

A basic Cocoa App will do: cocoa-app.png

Compile Cocoa App with myxcodebuild

Our myxcodebuild works just like xcodebuild.

Lets launch it within Cocoa App folder and do a clean build:

args.png cwd.png

Now we can watch our xcodebuild compiling our app while it’s being debugged inside Xcode 😃

debugging.png

And that’s it! Now you have your own personal xcodebuild and access to all of Xcode internals.

This article is just a first step in long series. I’m happy to help & support you with building new things in the languages we ❤️ most. Feel free to ping me on Twitter.

Happy programming!

Appendix: myxcodebuild on GitHub

https://github.com/roman-dzieciol/myxcodebuild

Appendix: Make Xcode project work in any location

Once frameworks are added, they will appear in the Xcode Project Navigator on the left.

By default paths to newly added frameworks will be relative to the project file, which will break if the project is moved.

We can make them relative to the Xcode.app Developer Directory so that it will work in any location.

relative-paths.png

Appendix: Complete Xcode3CommandLineBuildTool.h

Here is Xcode3CommandLineBuildTool.h in full:

#pragma once
@import Foundation;
@class Xcode3Project;
@class IDEWorkspace;
@class IDEXBSBuildParameters;
@class IDEScheme;
@class DVTMacroDefinitionTable;
@class DVTSourceControlWorkspaceBlueprint;
@class _TtC13IDEFoundation31IDEActivityLogEventStreamWriter;
@protocol IDEXBSXcodebuildSupportProvider;
@interface Xcode3CommandLineBuildTool : NSObject
@property int toolCommand;
@property (nonatomic) char shouldExit;
@property long long exitStatus;
@property (copy) NSString * projectName;
@property (retain) NSArray * targetNames;
@property (copy) NSString * workspaceName;
@property (copy) NSString * schemeName;
@property (copy) NSString * xcconfigPathFromOption;
@property (copy) NSString * xcconfigPathFromEnvVar;
@property (copy) NSString * actionResultsBundlePathWithBaselineOverridesFromOption;
@property (copy) NSString * automaticBaselineDescription;
@property (copy) NSString * nameOfFileToFind;
@property (copy) NSString * archivePath;
@property (copy) NSString * exportOptionsPlist;
@property (copy) NSString * exportDestinationPath;
@property (copy) NSString * buildMetricsPath;
@property (retain) NSArray * buildActions;
@property (retain) NSArray * potentialBuildActions;
@property (retain) NSArray * buildSettingAssignmentStrings;
@property (retain) Xcode3Project * project;
@property (retain) NSMutableArray * targets;
@property char allTargets;
@property (retain) IDEWorkspace * workspace;
@property (retain) NSMutableDictionary * perActionRunDestinations;
@property (copy) NSString * configurationName;
@property (retain) NSArray * architectures;
@property (copy) NSString * baseSdkName;
@property (retain) NSArray * toolchainNames;
@property (copy) NSArray * destinationSpecifications;
@property char skipUnsupportedDestinations;
@property (copy) NSNumber * destinationTimeout;
@property char parallelizeTargets;
@property char hideShellScriptEnvironment;
@property (copy) NSNumber * maxConcurrency;
@property (copy) NSNumber * maxDeviceTestConcurrency;
@property (copy) NSNumber * maxSimulatorTestConcurrency;
@property (retain) NSNumber * parallelTestingEnabledOverride;
@property (retain) NSNumber * parallelTestingWorkerCountOverride;
@property (retain) NSNumber * parallelTestingMaximumWorkerCount;
@property char dontActuallyRunCommands;
@property char skipUnavailableActions;
@property char quieterOutput;
@property (retain) NSString * localizationPath;
@property (retain) NSArray * exportLanguages;
@property (retain) NSString * codeCoverageEnabled;
@property (retain) NSString * localizableStringsDataEnabled;
@property (retain) NSString * addressSanitizerEnabled;
@property (retain) NSString * threadSanitizerEnabled;
@property (retain) NSString * UBSanitizerEnabled;
@property (retain) NSString * testRunSpecificationPathString;
@property (retain) NSArray * skipTestIdentifiers;
@property (retain) NSArray * onlyTestIdentifiers;
@property char runSkippedTestsOnly;
@property char disableConcurrentTesting;
@property (retain) NSMutableDictionary * testApplicationMappingOverrides;
@property (retain) NSString * testWithLanguage;
@property (retain) NSString * testWithRegion;
@property (retain) NSString * templateOutputPath;
@property (retain) NSString * templateTeamID;
@property (retain) NSString * templateName;
@property (retain) NSString * templatePlatform;
@property (retain) NSString * templateOptions;
@property (retain) NSDictionary * templateNonPermutedOptionValues;
@property (retain) NSArray * templateRequiredOptions;
@property (retain) NSString * clonedSourcePackagesDirPath;
@property char collectBuildTimeStatistics;
@property (retain) DVTMacroDefinitionTable * synthesizedMacros;
@property (retain) DVTMacroDefinitionTable * macrosFromCommandLine;
@property (retain) DVTMacroDefinitionTable * macrosFromXcconfigOption;
@property (retain) DVTMacroDefinitionTable * macrosFromXcconfigEnvVar;
@property (retain) NSMutableDictionary * userDefaults;
@property (retain) NSMutableDictionary * environmentUserDefaults;
@property (retain) NSOperationQueue * buildToolQueue;
@property (retain) NSString * resultBundlePath;
@property (copy) NSString * baseResultBundlePath;
@property (copy) NSString * sparseResultBundlePath;
@property (copy) NSString * extractOnlyTestIdentifier;
@property char outputAsJSON;
@property char readSourceControlBlueprint;
@property (retain) NSString * sourceControlSSHKeyPath;
@property (retain) DVTSourceControlWorkspaceBlueprint * sourceControlBlueprint;
@property char enableSourceControlKeychainAccess;
@property char disableSourceControlKeychainAccess;
@property (retain) _TtC13IDEFoundation31IDEActivityLogEventStreamWriter * activityLogStreamWriter;
@property (retain) IDEXBSBuildParameters * xbsBuildParameters;
@property (retain) NSObject<IDEXBSXcodebuildSupportProvider> * xbsXcodebuildSupportProvider;
@property char allowProvisioningUpdates;
@property char allowProvisioningDeviceRegistration;
@property (retain) IDEScheme * scheme;
@property (copy) NSString * name;
@property (copy) NSArray * arguments;
@property (copy) NSDictionary * environment;
@property (retain) NSFileHandle * standardInput;
@property (retain) NSFileHandle * standardOutput;
@property (retain) NSFileHandle * standardError;
@property (readonly) unsigned long long hash;
@property (readonly) Class superclass;
@property (readonly,copy) NSString * description;
@property (readonly,copy) NSString * debugDescription;
+ (char)useArchiveActionForInstall;
+ (char)enableInstallLocAction;
+ (id)timingLogAspect;
+ (id)xcodebuildDebugLogAspect;
+ (id)knownWorkspaceWrapperExtensions;
+ (id)filesInDirectory:(id)v1 withExtensions:(id)v2 errorString:(id *)v3;
+ (id)sharedCommandLineBuildTool;
- (id)init;
- (id)overridingProperties;
- (unsigned long long)_schemeLoadingTimeout;
- (unsigned long long)_projectLoadingTimeout;
- (void)_printWarningString:(id)v1;
- (void)_printErrorString:(id)v1 andFailWithCode:(long long)v2;
- (id)_supportedBuildActions;
- (id)_legacyBuildActionMapping;
- (id)_schemeCommandForBuildAction:(id)v1 outSchemeTask:(long long *)v2;
- (id)_actionStringForBuildAction:(id)v1;
- (void)_parseOptions;
- (id)_stringByResolvingSymlinksInPath:(id)v1;
- (id)_resolveSdk:(id)v1;
- (void)_resolveBaseSdk;
- (void)_resolveRunDestinationsForBuildAction:(id)v1;
- (id)_remainingUnavailableRunDestinationsAfterWaitingForDestinationsToBecomeAvailable:(id)v1;
- (id)_unavailableRunDestinationsInDestinations:(id)v1;
- (id)_availableDestinationsDescriptionForDestinations:(id)v1 scheme:(id)v2;
- (char)waitForRemoteSourcePackagesToFinishLoading;
- (void)_resolveInputOptionsWithTimingSection:(id)v1;
- (void)_workspace:(id)v1 failedToResolveContainerForProjectFile:(id)v2;
- (void)unableToOpenProjectAtPath:(id)v1 reason:(id)v2;
- (char)_shouldTestConcurrentlyForRunDestinations:(id)v1;
- (id)_concurrentTestOperations:(id)v1 schemeTask:(long long)v2 schemeCommand:(id)v3 executionEnvironment:(id)v4 invocationRecord:(id)v5 buildLog:(id)v6 restorePersistedBuildResults:(char)v7 deviceOperationLimit:(long long)v8 simulatorOperationLimit:(long long)v9 contextString:(id)v10 outError:(id *)v11;
- (void)_buildWithTimingSection:(id)v1;
- (void)_showBuildSettings;
- (void)_printVersionInfoAndExit;
- (id)schemeNamesInWorkspace:(id)v1;
- (void)_printContainerInformationAndExit;
- (id)_sdkForUseWithFind;
- (void)_printPathToExecutableAndExit;
- (void)_printPathToLibraryAndExit;
- (void)_enumerateAllPlatformsAlphabeticallyWithBlock:(void (^ /* unknown block signature */)(void))v1;
- (void)_writeJSONObjectTo:(id)v1 jsonObject:(id)v2;
- (id)_getJSONDataForSDK:(id)v1 platform:(id)v2;
- (char)_writeSDKListAsJSONTo:(id)v1;
- (void)_printShortSDKListAndExit;
- (void)_printDestinationListAndExit;
- (void)_printVerboseInfoForSDK:(id)v1 keysToEmit:(id)v2;
- (void)_printVerboseSDKListAndExit;
- (id)_availableExportArchiveOptionsSection;
- (void)_exportNotarizedAppAndExit;
- (void)_distributeArchiveAndExit;
- (void)_exportLocalizationsAndExit;
- (void)_importLocalizationsAndExit;
- (void)_resolvePackageDependenciesAndExit;
- (void)_saveProject;
- (void)_permuteTemplatesAndExit;
- (void)_createNewProjectAndExit;
- (void)_extractSparseResultBundle;
- (char)_shouldUseBuildMetricsFeature;
- (void)run;
- (long long)_buildLogVerbosity;
@end

I can’t wait to see what you will create 😊 And there’s so much more where that came from!


Roman Dzieciol

Written by Roman Dzieciol