Discard fastjson! Gson nanny level strategy for large scale project migration

Time:2021-6-3

preface

Hello, everyone, once again, I am going to enlarge dove’s three knife every day.

Before being taken off by everyone, I set a “lofty ideal” and must update the article this week. Now it seems that flag is useful…

This article is a summary of my efforts to help the group abandon the fastjson framework for more than a month. We have migrated most of the Java repositories from fastjson to gson.

The main reason for this is that the company is fed up with the frequent security vulnerabilities of fastjson. Every time there is a vulnerability, the company has to push a mandatory version upgrade of fastjson, which is a headache for the company.

In the first half of this article, I will briefly analyze the advantages and disadvantages of various JSON parsing frameworks, and give several solutions for enterprise level project migration of JSON framework.

In the second half of the article, I will summarize the problems of using gson and the pit that fastjson migrated to gson based on the experience of this month.

Contents:

  • Why give up fastjson?
  • Fastjson alternatives

    • Characteristics of three JSON frameworks
    • Performance comparison
    • Final choice
  • Considerations when replacing dependencies

    • Caution, caution, caution
    • Communication between development team and test team
    • Do regression / interface testing well
    • Consider the performance differences before and after migration
  • Replace fastjson with gson

    • JSON deserialization
    • Paradigm processing
    • List / map write
    • Hump and underline conversion
  • Migration FAQs

    • Date serialization is different
    • Springboot exception
    • Swagger exception
    • @Mapping jsonobject as input parameter exception

Note: whether to use fastjson is a controversial topic in recent years. This paper has no intention to discuss the right and wrong of framework selection, but only focuses on the problems encountered in migration. If you have any opinions you want to express, you can discuss them rationally in the comments section.

It takes about five minutes to read this article

Code word is not easy. Welcome to my personal official account: backend technology talks.

Why give up fastjson?

The reason is that,Fastjson vulnerability is frequentThis leads to the need to frequently urge all business lines to upgrade fastjson version to prevent security problems.

Fastjson frequently exposed security vulnerabilities in 2020. This vulnerability can bypass the autotype switch to achieve deserialization of remote code execution and obtain server access rights.

From v1.2.59 released in July 2019 to v1.2.71 released in June 2020, there are 13 official versions of autotype in each version.

Version history related to autotype in fastjson:

1.2.59 release to enhance the security when autotype is opened fastjson
1.2.60 release, added autotype blacklist, fixed denial of service security problem fastjson
1.2.61 release, adding autotype security blacklist fastjson
1.2.62, adding autotype blacklist, enhanced date deserialization and jsonpath fastjson
1.2.66 release, bug repair security reinforcement, and do security reinforcement, added autotype blacklist fastjson
1.2.67 release, bug repair security reinforcement, added autotype blacklist fastjson
1.2.68 release, support geojson, add autotype blacklist
1.2.69 released, fixed the new found high-risk autotype switch to bypass the security vulnerability, and added the autotype blacklist
1.2.70 released to improve compatibility and add autotype blacklist
1.2.71 release, supplement safety blacklist, no new use, preventive supplement

In contrast, other JSON frameworks, such as gson and Jackson, have fewer vulnerabilities,There are also fewer high-risk vulnerabilitiesThis is the main reason why companies want to replace the framework.

Fastjson alternatives

This paper mainly discusses the practical problem of replacing fastjson framework with gson, so we don’t discuss the advantages and disadvantages of various JSON frameworks in detail here, only give the conclusion.

After evaluation, there are mainly Jackson and gson frameworks under consideration, which are compared with fastjson.

Characteristics of three JSON frameworks

FastJson

Fast

Fastjson is faster than other JSON libraries. Since the release of 1.1. X version of fastjson in 2011, its performance has never been surpassed by other JSON libraries implemented by Java.

Widely used

Fastjson is widely used in Alibaba and deployed on tens of thousands of servers. Fastjson is widely accepted in the industry. In 2012, it was selected as one of the most popular domestic open source software by open source China.

The test is complete

Fastjson has many testcases. In version 1.2.11, there are more than 3321 testcases. Regression test will be conducted for each release to ensure stable quality.

Easy to use

Fastjson’s API is very simple.

Jackson

Easy to use – the Jackson API provides a high-level look and feel to simplify common use cases.

No need to create a mapping – the API provides the default mapping for most object serialization.

High performance – fast, low memory consumption, suitable for large object charts or systems.

Clean JSON – Jackson creates a clean and compact JSON result, which is easy to read.

No dependency – the library does not need any other libraries except JDK.

Gson

Provide a mechanism that makes it as easy to convert Java objects to JSON or vice versa as using toString () and a constructor (factory method).

Allow pre-existing immutable objects to be converted to JSON or vice versa.

Allows you to customize the representation of objects

Supports any complex object

Output lightweight and readable JSON

Performance comparison

Performance comparison source code written by colleagues:

https://github.com/zysrxx/jso…

This paper does not discuss the performance differences in detail. After all, it involves a lot of implementation ideas and optimizations of various frameworks

1. The performance of serializing single object is fastjson > Jackson > gson, in which the performance gap between fastjson and Jackson is very small, and the performance of gson is poor

2. The performance of serializing large objects is Jackson > fastjson > gson. When serializing large JSON objects, Jackson > gson > fastjson. When serializing big data, Jackson has obvious performance advantages

3. The performance of single object deserialization is fastjson > Jackson > gson

4. The performance of deserializing large objects is fastjson > Jackson > gson, and the performance gap is small

Final choice

  • Jackson is suitable for high performance scenarios, and gson is suitable for high security scenarios
  • For the new project repository, fastjson is no longer used. For stock system, considering the replacement cost of Jason, the following options are available:

    • The project does not use the autotype function, so it is recommended to switch directly to non fastjson. If the switching cost is large, you can consider continuing to use fastjson and closing safemode.
    • The business uses the autotype function, so it is recommended to promote the abandonment of fastjson.

Considerations for replacing dependencies

Characteristics of enterprise projects or large-scale projects:

  • The code structure is complex, and the team is maintained by many people.
  • To undertake important online business, serious bugs will lead to major accidents.
  • If it is an old project, there may be a lack of documents, which can not be modified at will.
  • There are many development branches in the project, which are constantly iterating online.

Therefore, for large projects, it is a complex and painful thing to migrate the underlying fastjson to gson. In fact, it is the same for the replacement of other dependencies.

I summarize the following issues that should be paid special attention to in the process of replacing project dependencies.

Caution, caution, caution

You can’t be too cautious. If the project you want to change is a very important business, once you make mistakes, the cost is very high. Moreover, for the business side and the product team, there is no new function online, but the system is blown up, which is an “unbearable” thing. Although you may feel aggrieved, because only you or your team know that although the business does not seem to have changed, the underlying code has changed dramatically.

So be careful!

Communication between development team and test team

In the process of dependency replacement, we need to do a good job in project planning, such as sub module replacement and strict subdivision scheduling.

The early planning, development and testing in order to work.

Between developers, we need to communicate the development considerations well in advance, such as the problem of depending on the version, so as to prevent multiple developers from modifying the code at the same time. Finally, we find that different versions are used, and the interface usage is different. This is very embarrassing, and it takes extra time to deal with.

For testing, we should communicate well in advance. Generally speaking, testing doesn’t care too much about technology projects that don’t change the business, because it’s neither optimizing speed nor new functions. But in fact, the migration involves the bottom layer, so it’s easy to get bugs. To let the test team understand that replacing project dependencies requires a lot of test time, and the cost is no less than new functions. Let them pay attention to it as much as possible.

Do regression / interface testing well

As mentioned above, the test team needs to invest a lot of man hours, which are mainly used for the overall regression of project functions, that is, regression testing.

Of course, it’s not just business regression testing. If possible, it’s interface regression testing.

If the company has an interface management platform, it can greatly improve the efficiency of this kind of project testing.

For example, after a module is modified, deploy an online version in the test environment (or sandbox environment), deploy a modified version, and directly compare the data returned by the interface. Generally speaking, it’s Jason comparison. There are a lot of tools for Jason comparison on the Internet

https://www.sojson.com/

Consider the performance differences before and after migration

Just as the performance comparison between gson and fastjson described above, the replacement framework needs to pay attention to the performance differences between frameworks, especially for traffic services, that is, high concurrency projects. If the response time changes greatly, it will cause the attention of upstream and downstream, leading to some additional consequences.

Replace fastjson with gson

This paper summarizes two common methods of JSON framework, and posts detailed code examples to help you quickly start gson and switch seamlessly!

JSON deserialization

String jsoncase = "[{\" ID \ ": 10001, \" date \ ": 1609316794600, \" name \ ": \" Xiao Ming \ "}, {\" ID \ ": 10002, \" date \ ": 1609316794600, \" name \ ": \" Xiao Li \ "}];

// fastjson
JSONArray jsonArray = JSON.parseArray(jsonCase);
System.out.println(jsonArray);
System.out.println(jsonArray.getJSONObject(0).getString("name"));
System.out.println(jsonArray.getJSONObject(1).getString("name"));
//Output:
//[{"date": 1609316794600, "name": "Xiao Ming", "Id": 10001}, {"date": 1609316794600, "name": "Xiao Li", "Id": 10002}]
//Xiao Ming
//Xiao Li

// Gson
JsonArray jsonArrayGson = gson.fromJson(jsonCase, JsonArray.class);
System.out.println(jsonArrayGson);
System.out.println(jsonArrayGson.get(0).getAsJsonObject().get("name").getAsString());
System.out.println(jsonArrayGson.get(1).getAsJsonObject().get("name").getAsString());
//Output:
//[{ID: 10001, date: 1609316794600, name: Xiaoming}, {ID: 10002, date: 1609316794600, name: Xiaoli}]
//Xiao Ming
//Xiao Li

It can be seen that the difference between the two is mainly in the various types of get. The gson calling method has changed, but not much.

Then, let’s see if there will be an exception in the deserialization of an empty object

String jsonObjectEmptyCase = "{}";

// fastjson
JSONObject jsonObjectEmpty = JSON.parseObject(jsonObjectEmptyCase);
System.out.println(jsonObjectEmpty);
System.out.println(jsonObjectEmpty.size());
//Output:
// {}
// 0

// Gson
JsonObject jsonObjectGsonEmpty = gson.fromJson(jsonObjectEmptyCase, JsonObject.class);
System.out.println(jsonObjectGsonEmpty);
System.out.println(jsonObjectGsonEmpty.size());
//Output:
// {}
// 0

Nothing unusual, happy.

Look at the empty array. After all, [] feels more error prone than {}.

String jsonArrayEmptyCase = "[]";

// fastjson
JSONArray jsonArrayEmpty = JSON.parseArray(jsonArrayEmptyCase);
System.out.println(jsonArrayEmpty);
System.out.println(jsonArrayEmpty.size());
//Output:
// []
// 0

// Gson
JsonArray jsonArrayGsonEmpty = gson.fromJson(jsonArrayEmptyCase, JsonArray.class);
System.out.println(jsonArrayGsonEmpty);
System.out.println(jsonArrayGsonEmpty.size());
//Output:
// []
// 0

The two frameworks have no problems, perfect analysis.

Paradigm processing

Parsing generics is a very common function. Most of the fastjson code in our project is parsing JSON and java bean.

//Entity class
User user = new User();
user.setId(1L);
User. Setusername ("Ma Yun");

// fastjson
List<User> userListResultFastjson = JSONArray.parseArray(JSON.toJSONString(userList), User.class);
List<User> userListResultFastjson2 = JSON.parseObject(JSON.toJSONString(userList), new TypeReference<List<User>>(){});
System.out.println(userListResultFastjson);
System.out.println("userListResultFastjson2" + userListResultFastjson2);
//Output:
//Userlistresultfastjson [user [hash = 483422889, id = 1, username = ma Yun], null]
//Userlistresultfastjson2 [user [hash = 488970385, id = 1, username = ma Yun], null]

// Gson
List<User> userListResultTrue = gson.fromJson(gson.toJson(userList), new TypeToken<List<User>>(){}.getType());
System.out.println("userListResultGson" + userListResultGson);
//Output:
//Userlistresultgson [user [hash = 1435804085, id = 1, username = ma Yun], null]

It can be seen that gson can also support generics.

List / map write

There is a difference between fastjson and gson. Gson does not support writing list to value directly, but fastjson does.

Therefore, gson can only parse the list and write it into value. See the following code for details:

//Entity class
User user = new User();
user.setId(1L);
User. Setusername ("Ma Yun");

// fastjson
JSONObject jsonObject1 = new JSONObject();
jsonObject1.put("user", user);
jsonObject1.put("userList", userList);
System.out.println(jsonObject1);
//Output:
//{"userlist": [{"Id": 1, "username": "Ma Yun"}, null], "user": {"Id": 1, "username": "Ma Yun"}}

// Gson
JsonObject jsonObject = new JsonObject();
jsonObject.add("user", gson.toJsonTree(user));
System.out.println(jsonObject);
//Output:
//{"user": {"Id": 1, "username": "Ma Yun"}, "userlist": [{"Id": 1, "username": "Ma Yun"}, null]}

In this way, gson doesn’t seem as convenient as fastjson, because the list is placed in thegson.toJsonTree(user)In the form of. In this way, you cannot enter the object first and modify it later( Some students are used to putting in objects first and then modifying them, so that the code will have to be changed.)

Hump and underline conversion

Hump conversion underline relies on modifying the gson serialization mode toLOWER_CASE_WITH_UNDERSCORES

GsonBuilder gsonBuilder = new GsonBuilder();
gsonBuilder.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES);
Gson gsonUnderScore = gsonBuilder.create();
System.out.println(gsonUnderScore.toJson(user));
//Output:
// {"id":1,"user_ Name ":" Ma Yun "}

Frequently asked questions

The following is a summary of the pits we stepped on in the process of the company’s project migration to gson. It seems that these pits have no technical content now. But that’s my original intention of writing this article, to help you avoid these difficult to find pits.

Some of these problems are found in the regression test, some in the self-test, and some after the launch, such as swagger hang up.

Date serialization is different

I don’t know if you’ve ever thought of a problem. If your project has a cache system and uses fastjson to write the cache, after you switch gson, you need to use gson to parse it. Therefore, it is necessary to ensure that the analytic logic of the two frameworks is the same, but obviously this wish is beautiful.

In the process of testing, we found the date type, which is resolved in different ways in the two frameworks.

  • Fastjson: date is directly parsed to UNIX
  • Gson: directly serialized to the standard format date

Discard fastjson! Gson nanny level strategy for large scale project migration

As a result, when gson deserializes the JSON, it directly reports an error and cannot be converted to date.

Solution:

Create a new class to resolve the date type

import com.google.gson.TypeAdapter;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonWriter;

import java.io.IOException;
import java.util.Date;

public class MyDateTypeAdapter extends TypeAdapter<Date> {
    @Override
    public void write(JsonWriter out, Date value) throws IOException {
        if (value == null) {
            out.nullValue();
        } else {
            out.value(value.getTime());
        }
    }

    @Override
    public Date read(JsonReader in) throws IOException {
        if (in != null) {
            return new Date(in.nextLong());
        } else {
            return null;
        }
    }
}

Next, when creating gson, put it into the special processing class as date:

Gson gson = new GsonBuilder().registerTypeAdapter(Date.class,new MyDateTypeAdapter()).create();

This allows gson to process the date as UNIX.

Of course, this is only for compatibility with the old cache. If you feel that your warehouse does not have this concern, you can ignore this problem.

Springboot exception

After switching to gson, the interface of the web project built with springboot cannot be requested directly. Error reporting is similar to:

org.springframework.http.converter.HttpMessageNotWritableException

Because the default mapper of springboot is Jackson parsing, after we switch to gson as the return object, Jackson can’t parse.

Solution:

Add in application.properties:

#Preferred JSON mapper to use for HTTP message conversion
spring.mvc.converters.preferred-json-mapper=gson

Swagger exception

This problem is similar to the spring boot exception above, because the introduction of gson in spring boot makes it impossible for swagger to parse JSON.

Discard fastjson! Gson nanny level strategy for large scale project migration

Adopt a solution similar to the following (add a gson adapter)

http://yuyublog.top/2018/09/0…

  1. GsonSwaggerConfig.java
@Configuration
public class GsonSwaggerConfig {
    //Set swagger to support gson
    @Bean
    public IGsonHttpMessageConverter IGsonHttpMessageConverter() {
        return new IGsonHttpMessageConverter();
    }
}
  1. IGsonHttpMessageConverter.java
public class IGsonHttpMessageConverter extends GsonHttpMessageConverter {
    public IGsonHttpMessageConverter() {
        //Custom gson adapter
        super.setGson(new GsonBuilder()
                .registerTypeAdapter(Json.class, new SpringfoxJsonToGsonAdapter())
                . serializenulls() // null values also participate in serialization
                .create());
    }
}
  1. SpringfoxJsonToGsonAdapter.java
public class SpringfoxJsonToGsonAdapter implements JsonSerializer<Json> {
    @Override
    public JsonElement serialize(Json json, Type type, JsonSerializationContext jsonSerializationContext) {
        return new JsonParser().parse(json.value());
    }
}

@Mapping jsonobject as input parameter exception

Sometimes, we use something like:

public ResponseResult<String> submitAudit(@RequestBody JsonObject jsonObject) {}

If you use this code, you actually use gson to parse the JSON string.But the risk is highIn general, please try to avoid using jsonobject to accept parameters directly.

In gson, if jsonobject has a number field, it will be serialized to double, that is, it willcount = 0This kind of sequence becomescount = 0.0

Why is this the case?In short, when gson resolves JSON to object type, it will use double conversion for numeric type by default.

If JSON corresponds to the object type, it will eventually resolve to the map < string, Object > type; The object type is related to the specific value in JSON. For example, the “” value in double quotation marks is translated into string. We can see that the number type is all converted to double type, so we have our previous problem. Integer data is translated to double type, for example, 30 becomes 30.0.

You can see the objecttypeadapter class of gson, which inherits the abstract class of gson’s typeadapter

Discard fastjson! Gson nanny level strategy for large scale project migration

For detailed source code analysis and principle elaboration, you can read this expanded reading:

https://www.jianshu.com/p/eaf…

Solution:

The first solution: accept the input parameter with entity class instead of jsonobject

The second solution: similar to the above solution to the date type problem, define an adapter to accept numbers and handle them. I think this idea is feasible, but it is difficult. It may affect other types of parsing, so we need to pay special attention when designing the adapter.

summary

This article is mainly for those students who need to migrate the project to the gson framework.

Generally speaking, personal small projects do not need to spend so much energy to do migration, so this article may target a narrow population.

But the article also mentioned a lotGeneral problemsFor example, how to evaluate the necessity of migration framework. We need to consider the framework compatibility, the performance difference between the two, the migration time and many other issues.

I hope this article can help you.

reference resources

How to migrate from fastjson to gson

https://juejin.im/post/684490…

The author of this article encapsulates the tool class to complete the migration

https://mxcall.github.io/post…

Do you really know how to use gson? Gson User Guide

https://www.jianshu.com/p/e74…

Performance comparison of JSON

https://github.com/zysrxx/jso…

Fastjson official document

https://github.com/alibaba/fa…

Yibai course

https://www.yiibai.com/jackson

Pay attention to me

I am an Internet back-end development engineer working in the front line.

Focus on back-end development, data security, edge computing and other directions, welcome to exchange.

I can be found on all major platforms

Main content of original article

  • Back end development (mainly Java)
  • Technical interview
  • Algorithm solution / data structure / design pattern
  • Interesting things in my life

Private official account: backend Technology

Discard fastjson! Gson nanny level strategy for large scale project migration

If the article is helpful to you, you may as well like it and collect it~