A recently published new patent takes Apple’s famous Apple Pencil up several notches in its abilities. Published 18 Jan 2018 in the US Patent Office, (patent no. 15650512) and filed last summer, the new vision for presumably a future Apple Pencil would enable it to be used as an input device without a touch-sensitive surface.
The Apple Pencil as Magic Wand
The patent shows how the invention would enable a future Apple Pencil to be used off of touch-sensitive surfaces—like the iPad Pro’s screen—and instead used on any type of surface or even in the air.
The abstract reads thus:
Content can be using an input device without a touch-sensitive surface. In some examples, touch-down and lift-off on a non-touch-sensitive surface can be monitored by a force sensor of the input device. The position and/or motion of the input device can be tracked according to various methods including one or more of a motion and orientation sensor, a camera, or an electromagnetic- or sound-based triangulation scheme. The force data and position/motion data can be processed to generate content, including textual character input and three-dimensional objects. In some examples, the content can be generated based on tracking position and/or motion of the input device without requiring contact with a surface.
Tracking of the location of the tip of input stylus would be by various means, including motion sensors on Apple devices, cameras, electromagnetic sensors and even sound-based triangulation schema. (see images 01 – 02)
01 – Figure 9 of the Apple Patent illustrates examples of configuration for generating content using an input device on a non-touch-sensitive surface.
Most importantly, as the abstract notes above, “touch-down and lift-off” on a non-touch-sensitive surface would be monitored by the force sensor in the stylus. In figure 9 from the patent application, a device with a touchscreen is shown (presume an iPad) and sensing hardware (RX1 – RXn) would be located in the device to help triangulate and position the input device (presume future Apple Pencil). (image 01 above)
As can be seen in figure 11 from the application below, even a laptop charge represents another location in space where another RX1 sensor could be located, helping increase the accuracy of positioning the input device in space. (see image 02)
02 – In figure 11 of the patent application, sensors are shown on devices, including a power charger, that helps locate the stylus in three dimensions.
What is important about this Apple patent application is that doesn’t just show future possibilities for the Apple Pencil but also suggest that the stylus can play an important role as an input device for other hardware beyond the iPad Pro—namely iMac and MacBook computers. (image 02)
Imagining the Impact on CAD and 3D
By locating the point of the future Apple Pencil in space the device could be used to draw in the air, like a magic wand, creating various types of input capacities. In very practical terms, the device could be used to physically trace actual physical objects—obviously of “non-touch-sensitive” surfaces. By using sound-based triangulation schema, the imagined future Apple Pencil could go around the back side of objects losing sight of camera sensors but maintaining sound-based contact with a larger device (iPad or even Mac computer). Take a look at the cylinder figure 13 from the patent filed images. (image 03)
03 – These patent images showcase the role of the future input device in such areas as creating a 3D model on the computer by tracing a physical object in space.
By tracing around the top and sides and bottom the height of the cylinder object is determined; by tracing around its sides the width and shape of the object are determined. This suggests that a future Apple Pencil could be used to trace over something truly complex like the human body or the body of an automobile and produce a 3D model of it. The multiple lines on the cylinder may suggest that a second pass around the cylinder helps ascertain its circumference.
04 – Today people use LDM (laser distance meters) to pair up with iPad and iPhone apps to measure physical spaces like room in buildings, but in this patent drawing, the future input device could do this faster.
Even more important to CAD and AEC professionals, in particular, a future Apple Pencil could be used to measure physical rooms—not just objects like cylinders—and thus eliminate or supplement the use of LDMs in building survey tasks. (image 04 above).
If Apple chose to pair a laser on the back-end of a future Apple Pencil, the device could be used to both grab points in the corner of a room (presumably the user doesn’t trace along the floor but rather moves the input device to the corner of the room and then either presses a button or utilizes the force sensor in the tip of the device) and also shoot a laser to grab points high up that are not accessible.
This would make it possible for the future Apple Pencil when paired with a Mac or iPad to do complex building surveys in the field, generating spatial models based on X, Y, and Z data points.
Architosh Analysis and Commentary
The iPad is already the most popular tablet device within the building industry in the US and some European countries. With the addition of an Apple Pencil as pictured above, many additional capabilities could come to popular apps used within the AEC industry on the iPad today.
Measuring rooms would very useful for just about all AEC pros using an iPhone or iPad. Adding the laser for reaching high places is only a bonus but not truly necessary.
Simply having the ability to draw on any surface, not just that of the iPad’s touch-sensitive surface, would be liberating. It could sponsor new types of larger surfaces on which to draw. The ability to draw in the air might spawn new art forms while tracing complex physical bodies adds new capacities and values to making physical models (model to 3D via Apple Pencil).
The Apple Pencil is already a stunning UX/UI device in the new Shapr3d app which we have profiled recently on Architosh. Letting that handheld stylus leave the limits of the iPad screen means freedom to work in more ways than one.