Microsoft Releases Kinect SDK 1.7 Enabling 3D Scanning Capability

Microsoft on March 18, 2013 launched the Kinect for Windows SDK v1.7. The new SDK includes the Kinect Fusion tool that enables the Kinect for Windows sensor to scan and create accurate 3D models. A 3D model produced by a Kinect sensor and the new software is illustrated below.

  Kinect Fusion enables developers to create accurate 3-D renderings in real time.
In announcing the new Kinect SDK Bob Heddle, Microsoft, Director Kinect for Windows, has described Kinect Fusion as one of the most affordable 3D scanning tools available today for creating 3D renderings of people and objects. Heddle goes on to say, “Kinect Fusion fuses together multiple snapshots from the Kinect for Windows sensor to create accurate, full, 3D models.

Developers can move a Kinect for Windows sensor around a person, object, or environment and “paint” a 3D image of the person or thing in real time. These 3D images can then be used to enhance countless real-world scenarios, including augmented reality, 3D printing, interior and industrial design, and body scanning for things such as improved clothes shopping experiences and better-fitting orthotics.”

Reporting from Microsoft Research’s annual TechFest event (Mar. 5-7, 2013, Microsoft Conference Center, Redmond, Washington, USA) IEEE Spectrum has posted a video in which Microsoft researchers describe and demonstrate 3D scanning using Kinect Fusion.





In the video, researcher Toby Sharp of Microsoft’s Cambridge UK group describes the operation of the commercial Kinect sensor with the new Kinect Fusion software as it scans and is able to “reconstruct the world in 3D.” The video also describes the challenge presented in processing the large amount of data required for this task at 30 frames per second in near real time as the Kinect camera is moving about the object being scanned.

The video illustrates that relatively detailed 3D scans and solid models of objects and people can be captured with good detail using the inexpensive Kinect sensor when combined with a good deal of processing power provided by a presumably high performance GPU (Graphic Processing Unit).

In a prior entry on Microsoft’s Kinect for Windows blog, the operation of Kinect Fusion is further described as “…taking the incoming depth data from the Kinect for Windows sensor and using the sequence of frames to build a highly detailed 3D map of objects or environments. The tool then averages the readings over hundreds or thousands of frames to achieve more detail than would be possible from just one reading. This allows Kinect Fusion to gather and incorporate data not viewable from any single view point. Among other things, it enables 3D object model reconstruction, 3D augmented reality, and 3D measurements. You can imagine the multitude of business scenarios where these would be useful, including 3D printing, industrial design, body scanning, augmented reality, and gaming.”

The business scenarios environed by Microsoft would seem to fit in well with the current interests of businesses and consumers in 3D printing. Moreover, the ability to quickly create accurate 3D solid models of objects, people and environments using relatively inexpensive equipment should open up a wide range of market applications. In releasing the new SDK, Microsoft has taken a large step toward enabling new opportunities for developers and end users.

By Phil Wright, Display Central

3D Spotlight

Three-dimensional movie production is about to enter a whole new world. The digital format already has proved its commercial appeal as films released this year in 3D have almost universally received a bump at the boxoffice. But as the Dec. 18 opening of James Cameron's inter-stellar adventure Avatar approaches, the 3D landscape is heating up on the technology side.

In part, this activity can be attributed to a recent increase in equipment manufacturers working to refine and expand 3D production capacity. Such manufacturers as P+S Technik, Element Technica and Binocle have developed 3D camera rigs, and it is now common to find 3D capabilities in leading postproduction systems, from companies including Avid, Autodesk, DVS, the Foundry and Quantel. Panasonic and Sony also are working on a full line of 3D products, aiming to provide the technology and a cost structure that makes 3D viable.

But the production of Avatar is shaping up as a key factor in the evolution of 3D, as the ambitious, $230 million-budgeted live-action/motion-capture film has become a testing ground for several technologies gaining traction in the 3D community.

For instance, the film's live action was lensed using the Fusion 3D camera system, which Cameron invented with Vince Pace, a director of photography on the film. First used to make Cameron's 2003 Imax 3D film Ghosts of the Abyss, the Fusion rigs -- which can be used with a variety of digital cinematography cameras -- are now available for rental via Pace's Burbank-based 3D provider, which continues to develop the system for other productions.

Fusion has been used on such live-action digital 3D titles as this summer's The Final Destination and concert films by Miley Cyrus and the Jonas Brothers. Those films benefited from the work done to prepare the camera system for the challenging Avatar production.

"This was a very ambitious film," Pace says. "They really became Avatar-specific rigs."

The movie, which is still not quite completed, is expected to be roughly 60% CG, including characters that were animated using new performance capture techniques; and 40% live action with a substantial amount of visual effects elements. Cameron shot the film in Hawaii and New Zealand, where Peter Jackson's Weta handled visual effects. The performance capture was done on a Marina Del Rey stage with the help of advanced technology company Giant Studios.

Insiders in the 3D community have been watching the production because of several innovations on display. The most significant new toy is an algorithm that guides the cinematographer through some of the mathematics of stereo vision.

"It essentially recognizes a focal length and adjusts the controls of the system to provide a starting point," Pace says. He believes this will allow the cinematographer to focus on creative decisions, rather than the technical.

"It will also help minimize the number of bad 3D shots," he adds. "With bad color, you walk away saying, 'That didn't look good.' In stereo, you walk away saying, 'That didn't feel good.' There is a big difference."

Pace and Cameron also tweaked the camera system to make it more manageable for director of photography Mauro Fiore. "The intention was not to have six cameras. It was to have three that could do six different configurations needed to get the job done," Pace explains. "We made some lens and some configuration choices to allow the camera -- this was a big change -- to flip so that whether we wanted it to be hand-held or on a dolly, it was as simple as just inverting the camera."

Meanwhile, Pace president Patrick Campbell developed a balance plate to keep the center of gravity consistent on a Steadicam during the course of the shot. "It was designed so that the Steadicam operator would not feel the movement of the camera rig during the course of a shot," Pace says.

Cameron also developed a new way to marry the performance capture and live-action production. The system, dubbed the Simulcam, allows the user of the Fusion camera system to look into the eyepiece and see the CG elements in real time. This way Cameron could operate the camera as if he was shooting from within the film's virtual world of Pandora.

Pace recalls the first time he was exposed to the Simulcam. It was during a scene in which the lead character Jake (Sam Worthington), a wounded Marine who travels to Pandora as an avatar and meets a race known as the Na'vi, was lying on the ground looking up at Neytiri (Zoe Saldana). Pace laid on the ground, pointed the camera upward, and looked into the eyepiece.

"I saw her there, she was looking at me as if I was Jake," he recalls. "I saw a knee in the shot, and I put my knee down, because I thought I was seeing my knee. But the knee was Jake's. From the minute you get behind that eyepiece, you become part of the interaction. It instantly fooled me; I was inside Jake's body."

The Avatar team believes the innovations created for the film will influence future productions because they allow filmmakers to forget about the science behind the creativity.

"It's not about technology; it's about philosophy," producer Jon Landau says. "It's more about a window into a world than a world coming out of a window. The goal of 3D is to duplicate human vision."

But Landau also hopes that this movie helps to prompt a shift in thinking about 3D.

"Our goal is to create the most engaging and immersive movie possible," he says. "I think 3D is something that helps you get there. The limiting factor for 3D has been the technology. With digital production you can now have high quality."

By Carolyn Giardina, The Hollywood Reporter

James Cameron Plugging 3D TV Sets

Titanic director James Cameron has signed on with Panasonic to promote new 3D TVs. The deal disclosed Friday comes as Cameron and Twentieth Century Fox are aiming to break new ground with the release of Avatar, a movie shot entirely in 3D.

At the same time, Panasonic is making a big push to get consumers excited about three-dimensional viewing in the home -- excited enough to buy new flat-panel sets and new Blu-ray disc players. Consumers will have to wear special glasses to experience the 3D effect.

Panasonic is planning to start selling 3D TVs next year. Rivals, including Sony, which has its own movie division, and Samsung Electronics of South Korea have shown prototypes and may offer similar products. It's not clear how much 3D TVs would cost.

The manufacturers face a problem in that 3D content is scarce. There's also no agreement on a disc or broadcast format to bring the content to TV sets, though the industry group behind the Blu-ray disc may be close to finalizing a standard.

"I believe 3D is how we will experience movies, gaming and computing in the near future. 3D is not something you watch. It's a reality you feel you could step into," Cameron said on video.

Panasonic is hoping its collaboration with Cameron will give its brand an edge as a 3D leader, and give the company ideas for technological improvements for home TVs, GM Masayuki Kozuka said.

"We want to get global interest rolling," he told the Associated Press. "For people to want to watch 3D at home, the movie has to be a blockbuster."

Panasonic plans to have several trailer-vans driving around in the U.S. and Europe next month with large-screen 3D TVs inside showing Avatar. In Japan, footage from Avatar -- a science-fiction romance set in a futuristic jungle inhabited by creatures evocative of Cameron's Aliens -- will appear in ads for 3D TVs. Cameron developed a new computer-controlled 3D camera system for the movie.

Source: The Hollywood Reporter

CNN's Holograms Not Really Holograms

CNN made waves on Tuesday night by incorporating three-dimensional holograms into its coverage of the U.S. election. The only problem was, they weren't really holograms.

"They were quite sophisticated, no doubt," said Hans Jürgen Kreuzer, a professor of theoretical physics at Dalhousie University and an expert on holography who watched the 3-D interviews. "But I immediately said to my wife that I don't think it has anything to do with holograms."

At about 7 p.m. EST, reporter Jessica Yellin, who was in Chicago, spoke with New York-based anchor Wolf Blitzer live "via hologram," CNN said. Yellin appeared somewhat fuzzy and her image, apparently projected a few feet in front of Blitzer, appeared to glow around the edges. "You're a terrific hologram," Blitzer said to her.

"It's like I follow the tradition of Princess Leia," she said, referring to the Star Wars character.

Yellin explained that her image was being filmed in Chicago by 35 high-definition cameras set in a ring inside a special tent, which were processed and synchronized by 20 computers to the cameras in the New York studio. The network, which made use of three-dimensional imaging technology produced by Norway-based Vizrt and Israel-based SportVu, billed the interview as a first for television. CNN also aired a second "hologram" interview between anchor Anderson Cooper and rapper Will.I.Am, who was also in Chicago.

The CNN anchors were not really speaking to three-dimensional projected images, but rather empty space, Kreuzer said. The images were simply added to what viewers saw on their screens at home, in much the same way computer-generated special effects are added to movies.

Kreuzer said the images were tomograms, which are images that are captured from all sides, reconstructed by computers, then displayed on screen. Holograms, on the other hand, are projected into space.

CNN officials could not be reached for comment.

Kreuzer said technology is not far from being able to produce what CNN had tried to do, although capturing and projecting holograms of big objects like people is still a ways off. Holographic images are generally captured and projected using coherent light such as lasers. A laser would need to be more than six feet in diameter to capture a person's image, which Kreuzer said is impossible because such a light would be blinding.

It may soon be possible to capture and project large objects using other sources of coherent light, such as light-emitting diodes. LEDs are considerably cheaper and safer than lasers, Kreuzer said. "There will be some rapid development now because of the cheapness of these LEDs," he said. "You can use a thousand if you want."

By Peter Nowak, CBC

Lifelike Animation Heralds New Era for Computer Games

Extraordinarily lifelike characters are to begin appearing in films and computer games thanks to a new type of animation technology developed by Image Metrics.

Click to watch the video
Emily - the woman in the above animation - was produced using a new modelling technology that enables the most minute details of a facial expression to be captured and recreated. She is considered to be one of the first animations to have overleapt a long-standing barrier known as "uncanny valley" - which refers to the perception that animation looks less realistic as it approaches human likeness.

Emily is a truly monumental achievement, recreating every nuance of human facial expression, even though what you’re actually looking at is the face of a digital actor. Created through a partnership with USC’s Institute for Creative Technologies (ICT), the team’s primary objective was to create a completely convincing, animated computer-generated face.

Using ICT’s special scanning system that can capture facial details down to the individual pore, the face of actress Emily O’Brien was transformed into a digital representation of herself, which could then be entirely machine-manipulated. A special spherical lighting rig captured O’Brien in 35 reference facial poses using a pair of high resolution digital cameras. The facial maps were then converted into 3D data using Image Metrics’ proprietary markerless motion capture technology.



"Ninety per cent of the work is convincing people that the eyes are real," Mike Starkenburg, chief operating officer of Image Metrics, said. "The subtlety of the timing of eye movements is a big one. People also have a natural asymmetry - for instance, in the muscles in the side of their face. Those types of imperfections aren't that significant but they are what makes people look real".

Previous methods for animating faces have involved putting dots on a face and observing the way the dots move, but Image Metrics analyses facial movements at the level of individual pixels in a video, meaning that the subtlest variations - such as the way the skin creases around the eyes, can be tracked.

"There's always been control systems for different facial movements, but say in the past you had a dial for controlling whether an eye was open or closed, and in one frame you set the eye at 3/4 open, the next 1/2 open etc. This is like achieving that degree of control with much finer movements. For instance, you could be controlling the movement in the top 3-4mm of the right side of the smile," Mr Starkenburg said.



For many years now, animators have come up against a barrier known as "uncanny valley", which refers to how, as a computer-generated face approaches human likeness, it begins take on a corpse-like appearance similar to that in some horror films. As a result, computer game animators have purposely simplified their creations so that the players realise immediately that the figures are not real.

"There came a point where animators were trying to create a face and there was a theory of diminishing returns," said Raja Koduri, chief technlology officer in graphics at AMD, the chip-maker. AMD last week released a new chip with a billion transistors that will be able to show off creations such as Emily by allowing a much greater number of computations per second. "If you're trying to process the graphics in a photo-realistic animation, in real-time, there's a lot of computation involved," said Mr Koduri. He said that AMD's new chip - the Radeon HD 4870 X2 - was able to process 2.4 teraflops of information per second, meaning it had a capability similar to a computer that - only 12 years ago - would have filled a room. AMD's chip fits inside a standard PC. But he said that the line between what was real and what was rendered would not be blurred completely until 2020.



There have been several advances in computer-generated imagery (CGI) in recent years. One project at the University of Southern California involves placing an actor inside a giant metallic orb which fires more than 3,000 lights from a range of different angles - and with different degrees of intensity - at the actor while he or she is are being filmed performing an action. The image captured by the camera can then be transported into another piece of film and the lighting effect (on the actor) chosen according to the ambient lighting in the scene.


Click to watch the video
Sources: Times Online (Jonathan Richards) and Technabob (Paul Strauss) via Marketsaw (Jim Dorey)

James Cameron's Avatar Creating Tech Buzz

With 17 months to go before the release of James Cameron's sci-fi epic Avatar, his first narrative feature since 1997's Titanic, anticipation already is enormous. The wildly ambitious project will be made in stereoscopic 3-D and combine live action and computer animation using visionary new filmmaking techniques.

Slated to open Dec. 18, 2009, the production already has been in the works for 2 1/2 years. When completed, Cameron expects Avatar to be about 60% CG animation, based on characters created using a newly developed performance capture-based process, and 40% live action, with a lot of VFX in the imagery.

"It is the most challenging film I've ever made," Cameron said.

Still, the innovative filmmaker and digital 3-D pioneer and champion has never shifted his emphasis from storytelling.

"You have to make a good film that would be a good film under any circumstances," he said. "You have to put the narrative first. The reality is no matter how many (3-D) screens we get, you are still going to have a large number of people -- possibly the majority of people -- who see the film in a 2-D environment."

The live-action principal photography for Avatar was shot in New Zealand last fall and winter using the Fusion 3-D camera system. Cameron first used the Fusion to make his 2003 Imax 3-D film Ghosts of the Abyss; he and Ghosts director of photography Vince Pace invented the camera system for the project.

Now, Fusion camera systems are available for rental via Burbank-based 3-D provider Pace, through which president Vince Pace and Cameron continue to innovate and develop the technology. The system already has made its mark, having been used on such pioneering live-action digital 3-D titles as Hannah Montana and Miley Cyrus: Best of Both Worlds Concert and Journey to the Center of the Earth.

Said Pace: "The systems themselves, in my opinion, can handle any creative challenge. We've learned a lot since shooting Ghosts of the Abyss."

With Avatar's principal photography completed, Cameron is focused on CG production. The helmer said his team has completed the performance capture (sometimes referred to as motion capture) of the actors and is in the post process of performance capture 3-D. The CG involved a large amount of additional R&D that afforded the director new creative options and flexibility. For one, the film used a new performance capture production workflow.

"The way we developed the performance capture workflow on Avatar is we have our virtual camera, which allows me to, in real time, hold a camera -- it's really a monitor -- in my hands and point it at the actors and see them as their CG chartacters," Cameron said.

The actors wear leotards and a "head rig" with a tiny standard-definition camera that takes an image of an actor's face. "That is going though facial algorithms and going back into the camera as a real-time CG face of the character," the helmer said. "You see it talk; you see the eyes move. It is pretty phenomenal.

"Once we've laid down a take, the take exists in the digital asset management system," he said. "It an be accessed at any time. Long after the actors have gone home, I'm still out there with the virtual camera, shooting coverage on the scene. I just have to play the take back. I can do the close up, the wide shot. I can even move them around on a limited basis. We relight it. We do all kinds of things.

"It's this amazing ability to quickly conjure scenes and images and great fantasyscapes that is very visual. We call it 'director centric' because I can use the camera to block the actors," Cameron related. "When you are doing performance capture, creatively it's very daunting. It's very hard to imagine what it will look like. But if you can see it, if you can have a virtual image of what is it going to be like, then you are there. As the processing power goes up our models get more sophisticated and our lighting tools get more sophisticated, even while we are making this movie. I'm still doing a lot of virtual camera work on the film ... on stuff that was shot six months ago."

Cameron also used what he calls FPR, or Facial Performance Replacement, which he likens to the film sound technique of ADR (Automated Dialogue Replacement). To describe the process, the director relates that he recently wanted to redo a line spoken by actor Laz Alonzo. "We changed the words and he redid the dialogue. We didn't have to recapture (his body performance) and he didn't have to put the performance capture suit on again. We were just creating new words, and we were creating a new face."

On the cinematography, Cameron related that his goal was to create "one movie where the aesthetics of physical production and the aesthetics of virtual production are, to the extent that we could do it, pretty much it identical."

Reaching this goal involved development of what Cameron calls the 'Simulcam,' which essentially treats a real camera like the virtual camera and in turn helps to remove guesswork. "We're taking our virtual production toolset and superimposing it on physical production," Cameron said. "We turned the set on the soundstage into a capture volume and turned the physical camera into a capture virtual camera, so we were able to integrate CG characters and environments into our live action."

As an example of how this works, he explained: "We have people in flying vehicles, and I can see what is outside the window, fed in, in real time."

On 3-D, both Cameron and Pace are looking ahead.

"The real question is 'where does all this go?" Cameron said. "Are we looking at a situation maybe 10-15 years out where most laptops are sold with 3-D stereoscopic screens, most monitors are stereo compatible, most DVD players can run stereo content? ... I can see this becoming much more pervasive that we are thinking now."

He and Pace believe content is the key.

Pace addressed one last--and not often addressed--aspect of 3-D: The archival value.

"I think back of our shots at Titanic (lensed for "Ghosts of the Abyss"). Those have incredible, future proof, archival value," Pace said. "When we look at (3-D) display devices in the home (which are already becoming available)--a lot of filmmakers and studios need to be making 3-D right now. Those production commitments are often based on the here and now, instead of thinking about how much value there is to this 3-D product in the future. Why not master in 3-D now if there is only an incremental expense? Why not think about that now?"

By Carolyn Giardina, The Hollywood Reporter

Radiohead's New Video: No Cameras, No Lights, Just Data

A few weeks ago we heard about a project Radiohead was working on. The band was making a new video, but they weren't using any cameras, just lasers and data. As you might imagine, we were intrigued.

The song is called “House of Cards,” from Radiohead’s recent “In Rainbows” album. In this new video, there were no cameras on set. Instead, two scanning technologies were used to capture 3D images. Geometric Informatics scanning systems produced structured light to capture 3D images at close proximity, while a Velodyne LIDAR system that uses multiple lasers was used to capture large environments such as landscapes. In the video, 64 lasers rotating and shooting in a 360 degree radius 900 times per minute produced all the exterior scenes.

Click to watch the video
Whether you're a music fan or a developer (or both), we agreed with the band that it would be great to give you a deeper look into how all of this was done, and even a chance to play with the data yourself, under a license that allows remixing. You can view the video, watch a short documentary about how it was made, interact with the video in 3D, download some of the data, and download an iGoogle theme and gadget - all here.


Making Of “House of Cards”
By Ola Rosling, The Official Google Blog