[Android] I used arcore to make a 1:1 up to

Time:2022-5-21

Recently, I saw a news that a 1:1 free GAODA settled in Jinqiao, Shanghai.

[Android] I used arcore to make a 1:1 up to
shanghai_gundam

As a Gundam lover, I always wanted to go to the scene to feel the real oppression of Gundam, but I never had the opportunity to go to Shanghai. However, this is not difficult for me. I made a 1:1 Gundam by myself with the help of AR technology

How about it? The effect is no worse than that of Shanghai Jinqiao~

What is ar (attributed reality)


AR (augmented reality) is an emerging technology in recent years. It can apply virtual elements such as 3D models to the real world after simulation. Virtual and reality information complement each other, so as to realize the “enhancement” of the real world.

Many people will confuse AR (augmented reality) with VR (virtual reality). The difference between the two lies in the proportion of virtual elements:

  • VR: the scenes and characters you see are all fake, which is to substitute your consciousness into a virtual world.
  • Ar: some of the scenes and characters seen are true and some are false. It is to bring virtual information into the real world.

VR technology has been studied for more than 30 years, while AR is much younger. It is gradually known by people with the popularity of smart phones and smart wearable devices. Compared with VR, the development threshold of AR is much lower. As long as there is a smartphone and with the help of arcore provided by Google, everyone can develop their own AR applications.

Google ARCore


ARCoreIt is an AR solution provided by Google, which provides an API for developers to perceive the surrounding environment and create an immersive ar experience through Android, IOS and other mobile platforms.

Arcore provides developers with three capabilities:

  • Motion tracking: the position of the photographer is tracked by identifying the visual feature points in the camera image, so as to determine the relative position change of the virtual elements
  • Environmental understanding: identify clusters of feature points on common horizontal or vertical surfaces (such as tables or walls), determine plane boundaries, and place virtual objects on planes
  • Ray prediction: predict the lighting conditions of the current scene. You can use this lighting information to illuminate the virtual ar object and simulate the shadow of the object in the real world

Sceneform SDK


Arcore provides AR with the ability to perceive the surrounding environment, but a complete ar application also needs to deal with the rendering of 3D models, which needs to be completed with the help of OpenGL es, and the learning cost is very high. Officials are aware of this problem. After the launch of arcore in 2017, it was launched at the IO conference in 2018SceneformThis 3D image rendering library on Android.

.obj , .fbxor.gltfAnd other common 3D model file formats, although they can be used in mainstream 3d software, in Android, we can only render them through opengl code. The sceneform can combine the model files in these formats with the dependent resource files(.mtl, .bin, .pngEtc.) convert to.sfaand.sfbFormat. The latter is a binary model file for scene form rendering, and the former is a readable summary file to describe the latter.

Compared with OpenGL, the use of sceneform is much simpler, andsfbYou can also preview the model in the IDE through the as plug-in provided by the sceneform.

Next, realize my 1:1 up to through sceneform and arcore

1. Add dependency to gradle

Create a new androidstudio project in the build.exe of root Add scene form plug-in to gradle

dependencies { 
    classpath 'com.google.ar.sceneform:plugin:1.17.1' 
}

Then in the app’s build AAR in gradle that relies on arcore and sceneform

dependencies {
    ...
    implementation 'com.google.ar:core:1.26.0'
    implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1'
    implementation 'com.google.ar.sceneform:core:1.17.1'
}

2. Manifest application permission

<uses-permission android:name="android.permission.CAMERA"/>

<!--  This app will only be visible to arcore enabled devices in Google play -- >
<uses-feature android:name="android.hardware.camera.ar" android:required="true"/>

<application …>
    …

  <!--  When installing app, if arcore is not installed on the device, Google play will automatically install -- >
  <meta-data android:name="com.google.ar.core" android:value="required" />

</application>

3. Layout documents

ARFragmentIt can be used to carry ar scenes and respond to user behavior. The simplest way to display virtual elements in Android is to add an arfragment to the layout:

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">
  
<fragment
        android:id="@+id/ux_fragment"
        android:name="com.google.ar.sceneform.ux.ArFragment"
        android:layout_width="match_parent"
        android:layout_height="0dp"
        android:layout_weight="9" />

</FrameLayout>

4. Make SFB model file

3D models are generally made by professional software such as Maya or 3D max. many 3D modeling lovers will upload their works to some design websites for free or paid download.

sketchfab_gundam

We can download 3D model files in common formats on the website. with.objAs an example, the obj file describes the vertex and fragment information of the polygon. In addition, the color, material and other information are stored in the supporting file.mtlIn the file, we will downloadobj/mtl/pngCopy the model files to the non assets directory, so as to avoid entering APK.

For example, APP / sampledata

We are building Gtadle passedsceneform.asset(...)add toobj > sfbThe configuration of the is as follows

sceneform.asset('sampledata/msz-006_zeta_gundam/scene.obj',
        'default',
        'sampledata/msz-006_zeta_gundam/scene.sfa',
        'src/main/assets/scene')

sampledata/msz-006_zeta_gundam/scene.objIs the location of the obj source file,src/main/assets/sceneIs the generated SFB target path. We generate the target file inassets/In, APK is entered to facilitate loading during operation.

After gradle is configured, sync and build the project. During the build process, you willassets/Generate SFB file with the same name

5. Load and render models

//MainActivity.kt

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        
        arFragment = supportFragmentManager.findFragmentById(R.id.ux_fragment) as ArFragment
        arFragment.setOnTapArPlaneListener { hitResult, plane, motionEvent ->
        
            if (plane.type != Plane.Type.HORIZONTAL_UPWARD_FACING ||
                    state != AnchorState.NONE) {
                [email protected]
            }

            val anchor = hitResult.createAnchor()
            placeObject(ux_fragment, anchor, Uri.parse("scene.sfb"))

        }
        
    }
        

Arfragment can respond to the user’s click behavior in the AR scene and add virtual elements to the click location,
Uri.parse("scene.sfb")It is used to obtain the model file generated in assets.

    private fun placeObject(fragment: ArFragment, anchor: Anchor, model: Uri) {
        ModelRenderable.builder()
                .setSource(fragment.context, model)
                .build()
                .thenAccept {
                    addNodeToScene(fragment, anchor, it)
                }
                .exceptionally { throwable : Throwable ->
                   Toast.makeText(arFragment.getContext(), "Error:$throwable.message", Toast.LENGTH_LONG).show();
                   [email protected] null
                }
    }
    

Provided by sceneformModelRenderableUsed for model rendering. adoptsetSourceLoad SFB model file

   private fun addNodeToScene(fragment: ArFragment, anchor: Anchor, renderable: Renderable) {
        val anchorNode = AnchorNode(anchor)
        val node = TransformableNode(fragment.transformationSystem)
        node.renderable = renderable
        node.setParent(anchorNode)
        fragment.arSceneView.scene.addChild(anchorNode)
        node.select()
    }

ARSceneViewHold oneScene, scene is a tree data structure. As the root node of AR scene, various virtual elements will be added to the scene as their child nodes for rendering

val node = TransformableNode(fragment.transformationSystem)
node.renderable = renderable
node.setParent(anchorNode)

Therefore, rendering a 3D model is actually adding a node and setting itRenderableThe process of.

hitResultIs the location information clicked by the user,AnchorCreate an anchor point based on hitresult, which is added to the scene root node as a child node, and at the same time asTransformableNodeThe parent node of the. Transformablenode is used to carry the 3D model. It can accept gestures to drag or zoom in and out. Adding it to arch is equivalent to placing the 3D model at the click position.

6. Complete code

class MainActivity : AppCompatActivity() {
    lateinit var arFragment: ArFragment
    
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        if (!checkIsSupportedDeviceOrFinish(this)) return
        setContentView(R.layout.activity_main)
        
        arFragment = supportFragmentManager.findFragmentById(R.id.ux_fragment) as ArFragment
        arFragment.setOnTapArPlaneListener { hitresult: HitResult, plane: Plane, motionevent: MotionEvent? ->
            if (plane.type != Plane.Type.HORIZONTAL_UPWARD_FACING)
                [email protected]
            val anchor = hitresult.createAnchor()
            placeObject(arFragment, anchor, R.raw.cube)
        }
    }

    private fun placeObject(arFragment: ArFragment, anchor: Anchor, uri: Int) {
        ModelRenderable.builder()
                .setSource(arFragment.context, uri)
                .build()
                .thenAccept { modelRenderable: ModelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable) }
                .exceptionally { throwable: Throwable ->
                   Toast.makeText(arFragment.getContext(), "Error:$throwable.message", Toast.LENGTH_LONG).show();
                    [email protected] null
                }
    }

    private fun addNodeToScene(arFragment: ArFragment, anchor: Anchor, renderable: Renderable) {
        val anchorNode = AnchorNode(anchor)
        val node = TransformableNode(arFragment.transformationSystem)
        node.renderable = renderable
        node.setParent(anchorNode)
        arFragment.arSceneView.scene.addChild(anchorNode)
        node.select()
    }

    private fun checkIsSupportedDeviceOrFinish(activity: Activity): Boolean {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
            Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show()
            activity.finish()
            return false
        }
        val openGlVersionString = (activity.getSystemService<Any>(Context.ACTIVITY_SERVICE) as ActivityManager)
                .deviceConfigurationInfo
                .glEsVersion
        if (openGlVersionString.toDouble() < MIN_OPENGL_VERSION) {
            Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
                    .show()
            activity.finish()
            return false
        }
        return true
    }

    companion object {
        private const val MIN_OPENGL_VERSION = 3.0
    }
}

checkIsSupportedDeviceOrFinishIt is used to detect the runnable environment. Through the implementation, it can be seen that the running conditions of sceneform are androidn and OpenGL 3.0 or above.

The above is all the code. Although there are few codes, the effect is very wow

ar zeta gundam

last

With arcore, sceneform can quickly build AR applications. In addition to loading static 3D models, sceneform can also load animated models.

With the rise of the concept of “meta universe”, giants such as Google and Facebook will increase their research investment in AR and even VR technology. Virtual reality technology may become a new generation of social and entertainment scenes after mobile Internet, with huge imagination space.

That’s all for today. I’m going to play with my little sister I just met

ar girl

Finally, I recommend a website where you can download some interesting 3D models~

https://sketchfab.com/