Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce UTM Remote client for iOS/visionOS #6115

Merged
merged 75 commits into from
Feb 26, 2024
Merged
Changes from 1 commit
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
4706e22
project: rename preprocessor identifier for conditional features
osy Oct 29, 2023
6155ea8
project: add UTM Remote target
osy Oct 29, 2023
26ceb8b
project: rename scheme iOS-TCI to iOS-SE
osy Oct 29, 2023
ac2d7c4
github: configure build for iOS-Remote
osy Oct 29, 2023
7985cde
remote: implement key manager
osy Jan 24, 2024
1bd1b5f
remote: add remote server
osy Jan 24, 2024
9f5a586
remote(server): add server view
osy Jan 24, 2024
54acd99
remote: add remote client
osy Jan 24, 2024
50ef326
remote(client): add connect view
osy Jan 24, 2024
932ded4
github: update to Xcode 15.2
osy Jan 24, 2024
e2e827b
remote: establish handshake between client and server
osy Jan 27, 2024
75bb06f
vm(qemu): refactor SPICE related functionality to separate protocol
osy Jan 29, 2024
80b88bf
remote: implement listing and displaying VMs
osy Jan 29, 2024
58df784
remote: hide QEMU internals from remote SPICE VM
osy Jan 29, 2024
6414dcc
data: implement remote session management
osy Feb 4, 2024
894a691
vm(qemu): implement starting with remote SPICE client
osy Feb 4, 2024
d2d1418
remote: implement start remote vm
osy Feb 4, 2024
0ace64f
spice: support connecting from socket
osy Feb 5, 2024
be1f4ec
project: separate Info.plist for UTM Remote
osy Feb 5, 2024
a258ec3
project: remove UTMQemuVirtualMachine from Remote client builds
osy Feb 5, 2024
f5169c8
build: strip broken entitlements from ANGLE signature
osy Feb 5, 2024
545e36b
remote: implement client side connection
osy Feb 5, 2024
27103a9
remote: implement other VM actions
osy Feb 6, 2024
4fada3a
settings: exclude non-compatible settings for iOS SE and Remote builds
osy Feb 6, 2024
67f127a
project: remove QEMUKit dependency from iOS-Remote
osy Feb 6, 2024
6c9a660
remote: rework how client state transitions are handled
osy Feb 9, 2024
dbdf749
remote: send custom icon
osy Feb 9, 2024
ea958e6
remote: support SPICE TLS
osy Feb 11, 2024
1afd66a
home(visionOS): fix crash
osy Feb 11, 2024
745cd38
remote: add SPICE ticket password auth
osy Feb 11, 2024
fd4f173
connect: show model of Mac
osy Feb 11, 2024
38b5144
remote(client): implement fingerprint verification and improved conne…
osy Feb 12, 2024
427a201
remote(client): support connecting to specified host and port
osy Feb 12, 2024
eda9e94
remote: add fingerprint verification for client and server
osy Feb 12, 2024
d966e1c
preferences: add server settings
osy Feb 12, 2024
c3939e3
remote: add password authentication
osy Feb 12, 2024
a3b0c76
remote(client): implement timeout for connect attempt
osy Feb 13, 2024
ac5842c
remote(server): implement autostart, autoblock, and specify port
osy Feb 13, 2024
dd4de86
remote(server): add automatic NAT configuration
osy Feb 14, 2024
3232168
remote: support NAT mapping of SPICE port
osy Feb 14, 2024
c9927a3
connect: change trust button back to connect when entering password
osy Feb 14, 2024
db038df
vm(qemu): if serverPort is set, allocate unused port after it
osy Feb 16, 2024
452f8c2
remote: implement ReorderVirtualMachines and GetPackageSize
osy Feb 16, 2024
4c910d8
home: disable features not implemented for remote yet
osy Feb 16, 2024
1bed953
remote: sync home view between multiple clients and server
osy Feb 18, 2024
324f981
keyboard(iOS): disable logging for privacy reasons
osy Feb 18, 2024
2538c20
vm(remote): avoid state update collisions
osy Feb 18, 2024
5f7e11e
display(visionOS): dynamic resolution from window resize
osy Feb 21, 2024
45d216a
config(qemu): fix invalid GPU remapping for remote
osy Feb 21, 2024
8654fb3
remote: fix duplicate list changed events
osy Feb 21, 2024
8c88fd9
display(iOS): make QEMU errors non-fatal
osy Feb 21, 2024
689367a
home(iOS): support multiple sessions
osy Feb 21, 2024
9a871c0
display(visionOS): disable background auto-suspend on visionOS
osy Feb 21, 2024
bc30839
remote: do not discard saved password on connect
osy Feb 21, 2024
8ea2fb4
display(iOS): fix crash due to race when re-sizing while a view is be…
osy Feb 21, 2024
f812ab2
toolbar(iOS): disable removable drives from remote clients
osy Feb 21, 2024
07650fa
remote: fix continuation bug
osy Feb 21, 2024
da9c5c4
vm: get screenshot PNG data early
osy Feb 21, 2024
e6653fd
remote: add mount support tools command
osy Feb 21, 2024
d83fedf
vmdata: fix memory leak
osy Feb 23, 2024
f2f9db1
vm(qemu): fix hang when vm was improperly stopped
osy Feb 23, 2024
31ebc6f
remote: use separate queue for handling connections
osy Feb 23, 2024
52a1f45
session: force kill vm when multiple VMs are supported
osy Feb 24, 2024
0a8bff6
data: busyWorkAsync should return the task
osy Feb 24, 2024
4dca247
remote: re-connect when server is disconnected
osy Feb 24, 2024
f446c1c
vm(remote): fix memory leak
osy Feb 24, 2024
b762149
vm(remote): handle SPICE disconnect
osy Feb 24, 2024
51a7969
remote: support takeover of existing session and auto-pause of orphan…
osy Feb 25, 2024
3a57588
vm(remote): keep existing VM object when refreshing list
osy Feb 25, 2024
7b94235
display(visionOS): disable GCMouse due to it not working
osy Feb 25, 2024
1a966d2
home(visionOS): use plain window style
osy Feb 25, 2024
2947306
display(iOS): do not stop session until after popup is dismissed
osy Feb 25, 2024
e4dab5d
display(visionOS): disable hidden cursor because it is broken
osy Feb 25, 2024
9835e2b
project: update dependencies
osy Feb 25, 2024
aa071bd
display(visionOS): integrate VisionKeyboardKit
osy Feb 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
display(visionOS): dynamic resolution from window resize
osy committed Feb 25, 2024
commit 5f7e11e161c18a628173f335aba9aa0753fc046a
2 changes: 2 additions & 0 deletions Platform/iOS/Display/VMDisplayMetalViewController.h
Original file line number Diff line number Diff line change
@@ -42,6 +42,8 @@ NS_ASSUME_NONNULL_BEGIN

@property (nonatomic, strong) NSMutableArray<UIKeyCommand *> *mutableKeyCommands;

@property (nonatomic) BOOL isDynamicResolutionSupported;

- (instancetype)initWithCoder:(NSCoder *)coder NS_UNAVAILABLE;
- (instancetype)initWithNibName:(nullable NSString *)nibNameOrNil bundle:(nullable NSBundle *)nibBundleOrNil NS_UNAVAILABLE;
- (instancetype)initWithDisplay:(CSDisplay *)display input:(nullable CSInput *)input NS_DESIGNATED_INITIALIZER;
121 changes: 88 additions & 33 deletions Platform/iOS/Display/VMDisplayMetalViewController.m
Original file line number Diff line number Diff line change
@@ -29,11 +29,15 @@
#import "UTM-Swift.h"
@import CocoaSpiceRenderer;

static const NSInteger kResizeDebounceSecs = 1;
static const NSInteger kResizeTimeoutSecs = 5;

@interface VMDisplayMetalViewController ()

@property (nonatomic, nullable) CSMetalRenderer *renderer;
@property (nonatomic) CGFloat windowScaling;
@property (nonatomic) CGPoint windowOrigin;
@property (nonatomic, nullable) id debounceResize;
@property (nonatomic, nullable) id cancelResize;
@property (nonatomic) BOOL ignoreNextResize;

@end

@@ -43,9 +47,6 @@ - (instancetype)initWithDisplay:(CSDisplay *)display input:(CSInput *)input {
if (self = [super initWithNibName:nil bundle:nil]) {
self.vmDisplay = display;
self.vmInput = input;
self.windowScaling = 1.0;
self.windowOrigin = CGPointZero;
[self addObserver:self forKeyPath:@"vmDisplay.displaySize" options:NSKeyValueObservingOptionNew context:nil];
}
return self;
}
@@ -128,22 +129,26 @@ - (void)viewWillDisappear:(BOOL)animated {
[super viewWillDisappear:animated];
[self stopGCMouse];
[self.vmDisplay removeRenderer:self.renderer];
[self removeObserver:self forKeyPath:@"vmDisplay.displaySize"];
}

- (void)viewDidAppear:(BOOL)animated {
[super viewDidAppear:animated];
self.delegate.displayViewSize = [self convertSizeToNative:self.view.bounds.size];
[self addObserver:self forKeyPath:@"vmDisplay.displaySize" options:(NSKeyValueObservingOptionNew | NSKeyValueObservingOptionInitial) context:nil];
}

- (void)viewWillTransitionToSize:(CGSize)size withTransitionCoordinator:(id<UIViewControllerTransitionCoordinator>)coordinator {
[super viewWillTransitionToSize:size withTransitionCoordinator:coordinator];
[coordinator animateAlongsideTransition:nil completion:^(id<UIViewControllerTransitionCoordinatorContext> _Nonnull context) {
self.delegate.displayViewSize = [self convertSizeToNative:size];
[self.delegate display:self.vmDisplay didResizeTo:self.vmDisplay.displaySize];
if (self.delegate.qemuDisplayIsDynamicResolution && self.isDynamicResolutionSupported) {
if (!CGSizeEqualToSize(size, self.vmDisplay.displaySize)) {
[self requestResolutionChangeToSize:size];
}
}
}];
if (self.delegate.qemuDisplayIsDynamicResolution) {
[self displayResize:size];
}
}

- (void)enterSuspendedWithIsBusy:(BOOL)busy {
@@ -161,8 +166,8 @@ - (void)enterLive {
[super enterLive];
self.prefersPointerLocked = YES;
self.view.window.isIndirectPointerTouchIgnored = YES;
if (self.delegate.qemuDisplayIsDynamicResolution) {
[self displayResize:self.view.bounds.size];
if (self.delegate.qemuDisplayIsDynamicResolution && self.isDynamicResolutionSupported) {
[self requestResolutionChangeToSize:self.view.bounds.size];
}
if (self.delegate.qemuHasClipboardSharing) {
[[UTMPasteboard generalPasteboard] requestPollingModeForObject:self];
@@ -200,11 +205,21 @@ - (CGSize)convertSizeToNative:(CGSize)size {
return size;
}

- (void)displayResize:(CGSize)size {
UTMLog(@"resizing to (%f, %f)", size.width, size.height);
size = [self convertSizeToNative:size];
CGRect bounds = CGRectMake(0, 0, size.width, size.height);
[self.vmDisplay requestResolution:bounds];
- (void)requestResolutionChangeToSize:(CGSize)size {
self.debounceResize = [self debounce:kResizeDebounceSecs context:self.debounceResize action:^{
UTMLog(@"DISPLAY: requesting resolution (%f, %f)", size.width, size.height);
CGSize newSize = [self convertSizeToNative:size];
CGRect bounds = CGRectMake(0, 0, newSize.width, newSize.height);
self.debounceResize = nil;
#if defined(TARGET_OS_VISION) && TARGET_OS_VISION
self.cancelResize = [self debounce:kResizeTimeoutSecs context:self.cancelResize action:^{
self.cancelResize = nil;
UTMLog(@"DISPLAY: requesting resolution cancelled");
[self resizeWindowToDisplaySize];
}];
#endif
[self.vmDisplay requestResolution:bounds];
}];
}

- (void)setVmDisplay:(CSDisplay *)display {
@@ -217,8 +232,6 @@ - (void)setVmDisplay:(CSDisplay *)display {

- (void)setDisplayScaling:(CGFloat)scaling origin:(CGPoint)origin {
self.vmDisplay.viewportOrigin = origin;
self.windowScaling = scaling;
self.windowOrigin = origin;
if (!self.delegate.qemuDisplayIsNativeResolution) {
scaling = CGPointToPixel(scaling);
}
@@ -229,25 +242,67 @@ - (void)setDisplayScaling:(CGFloat)scaling origin:(CGPoint)origin {

- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary<NSKeyValueChangeKey,id> *)change context:(void *)context {
if ([keyPath isEqualToString:@"vmDisplay.displaySize"]) {
#if defined(TARGET_OS_VISION) && TARGET_OS_VISION
dispatch_async(dispatch_get_main_queue(), ^{
CGSize minSize = self.vmDisplay.displaySize;
if (self.delegate.qemuDisplayIsNativeResolution) {
minSize.width = CGPixelToPoint(minSize.width);
minSize.height = CGPixelToPoint(minSize.height);
UTMLog(@"DISPLAY: vmDisplay.displaySize changed");
if (self.cancelResize) {
[self debounce:0 context:self.cancelResize action:^{}];
self.cancelResize = nil;
}
self.debounceResize = [self debounce:kResizeDebounceSecs context:self.debounceResize action:^{
[self resizeWindowToDisplaySize];
}];
}
}

- (void)setIsDynamicResolutionSupported:(BOOL)isDynamicResolutionSupported {
if (_isDynamicResolutionSupported != isDynamicResolutionSupported) {
_isDynamicResolutionSupported = isDynamicResolutionSupported;
UTMLog(@"DISPLAY: isDynamicResolutionSupported = %d", isDynamicResolutionSupported);
if (self.delegate.qemuDisplayIsDynamicResolution) {
if (isDynamicResolutionSupported) {
[self requestResolutionChangeToSize:self.view.bounds.size];
} else {
[self resizeWindowToDisplaySize];
}
CGSize displaySize = CGSizeMake(minSize.width * self.windowScaling, minSize.height * self.windowScaling);
CGSize maxSize = CGSizeMake(UIProposedSceneSizeNoPreference, UIProposedSceneSizeNoPreference);
UIWindowSceneGeometryPreferencesVision *geoPref = [[UIWindowSceneGeometryPreferencesVision alloc] initWithSize:displaySize];
geoPref.minimumSize = minSize;
geoPref.maximumSize = maxSize;
geoPref.resizingRestrictions = UIWindowSceneResizingRestrictionsUniform;
[self.view.window.windowScene requestGeometryUpdateWithPreferences:geoPref errorHandler:nil];
});
}
}
}

- (void)resizeWindowToDisplaySize {
CGSize displaySize = self.vmDisplay.displaySize;
UTMLog(@"DISPLAY: request window resize to (%f, %f)", displaySize.width, displaySize.height);
#if defined(TARGET_OS_VISION) && TARGET_OS_VISION
CGSize minSize = displaySize;
if (self.delegate.qemuDisplayIsNativeResolution) {
minSize.width = CGPixelToPoint(minSize.width);
minSize.height = CGPixelToPoint(minSize.height);
}
CGSize maxSize = CGSizeMake(UIProposedSceneSizeNoPreference, UIProposedSceneSizeNoPreference);
UIWindowSceneGeometryPreferencesVision *geoPref = [[UIWindowSceneGeometryPreferencesVision alloc] initWithSize:minSize];
if (self.delegate.qemuDisplayIsDynamicResolution && self.isDynamicResolutionSupported) {
geoPref.minimumSize = CGSizeMake(800, 600);
geoPref.maximumSize = maxSize;
geoPref.resizingRestrictions = UIWindowSceneResizingRestrictionsFreeform;
} else {
geoPref.minimumSize = minSize;
geoPref.maximumSize = maxSize;
geoPref.resizingRestrictions = UIWindowSceneResizingRestrictionsUniform;
}
dispatch_async(dispatch_get_main_queue(), ^{
CGSize currentViewSize = self.view.bounds.size;
UTMLog(@"DISPLAY: old view size = (%f, %f)", currentViewSize.width, currentViewSize.height);
if (CGSizeEqualToSize(minSize, currentViewSize)) {
// since `-viewWillTransitionToSize:withTransitionCoordinator:` is not called
self.delegate.displayViewSize = [self convertSizeToNative:currentViewSize];
[self.delegate display:self.vmDisplay didResizeTo:displaySize];
}
[self.view.window.windowScene requestGeometryUpdateWithPreferences:geoPref errorHandler:nil];
});
#else
[self.delegate display:self.vmDisplay didResizeTo:self.vmDisplay.displaySize];
#endif
if (CGSizeEqualToSize(displaySize, CGSizeZero)) {
return;
}
[self.delegate display:self.vmDisplay didResizeTo:displaySize];
#endif
}

@end
11 changes: 11 additions & 0 deletions Platform/iOS/Display/VMDisplayViewController.swift
Original file line number Diff line number Diff line change
@@ -134,4 +134,15 @@ public extension VMDisplayViewController {
func integerForSetting(_ key: String) -> Int {
return UserDefaults.standard.integer(forKey: key)
}

@discardableResult
func debounce(_ delaySeconds: Int, context: Any? = nil, action: @escaping () -> Void) -> Any {
if context != nil {
let previous = context as! DispatchWorkItem
previous.cancel()
}
let item = DispatchWorkItem(block: action)
DispatchQueue.main.asyncAfter(deadline: .now() + .seconds(delaySeconds), execute: item)
return item
}
}
1 change: 1 addition & 0 deletions Platform/iOS/VMDisplayHostedView.swift
Original file line number Diff line number Diff line change
@@ -190,6 +190,7 @@ struct VMDisplayHostedView: UIViewControllerRepresentable {
}
// some obscure SwiftUI error means we cannot refer to Coordinator's state binding
vc.setDisplayScaling(state.displayScale, origin: state.displayOrigin)
vc.isDynamicResolutionSupported = state.isDynamicResolutionSupported
}
case .serial(let serial, _):
if let vc = uiViewController as? VMDisplayTerminalViewController {
10 changes: 9 additions & 1 deletion Platform/iOS/VMSessionState.swift
Original file line number Diff line number Diff line change
@@ -78,7 +78,9 @@ import SwiftUI
@Published var externalWindowBinding: Binding<VMWindowState>?

@Published var hasShownMemoryWarning: Bool = false


@Published var isDynamicResolutionSupported: Bool = false

private var hasAutosave: Bool = false

init(for vm: any UTMSpiceVirtualMachine) {
@@ -291,6 +293,12 @@ extension VMSessionState: UTMSpiceIODelegate {
}
}
#endif

nonisolated func spiceDynamicResolutionSupportDidChange(_ supported: Bool) {
Task { @MainActor in
isDynamicResolutionSupported = supported
}
}
}

#if WITH_USB
2 changes: 2 additions & 0 deletions Platform/iOS/VMWindowState.swift
Original file line number Diff line number Diff line change
@@ -71,6 +71,8 @@ struct VMWindowState: Identifiable {
var isRunning: Bool = false

var alert: Alert?

var isDynamicResolutionSupported: Bool = false
}

// MARK: - VM action alerts
3 changes: 3 additions & 0 deletions Platform/iOS/VMWindowView.swift
Original file line number Diff line number Diff line change
@@ -171,6 +171,9 @@ struct VMWindowView: View {
.onChange(of: session.vmState) { [oldValue = session.vmState] newValue in
vmStateUpdated(from: oldValue, to: newValue)
}
.onChange(of: session.isDynamicResolutionSupported) { newValue in
state.isDynamicResolutionSupported = newValue
}
.onReceive(keyboardDidShowNotification) { _ in
state.isKeyboardShown = true
state.isKeyboardRequested = true