In depth explanation of spring boot efficient data aggregation

Time:2019-12-6

background

Interface development is the most common scenario in back-end development, which may be restful interface or RPC interface. Interface development is often to extract data from everywhere and then assemble the results, especially those partial business interfaces

For example, I need to implement an interface to pull the integration data of user basic information + user’s blog list + user’s fan data. Suppose there are three interfaces that can be used to obtain user basic information, user’s blog list and user’s fan data

Basic user information


@Service
public class UserServiceImpl implements UserService {
 @Override
 public User get(Long id) {
 try {Thread.sleep(1000L);} catch (InterruptedException e) {}
 /* mock a user*/
 User user = new User();
 user.setId(id);
 user.setEmail("[email protected]");
 user.setUsername("lvyahui8");
 return user;
 }
}

User blog list


@Service
public class PostServiceImpl implements PostService {
 @Override
 public List<Post> getPosts(Long userId) {
 try { Thread.sleep(1000L); } catch (InterruptedException e) {}
 Post post = new Post();
 post.setTitle("spring data aggregate example");
 post.setContent("No active profile set, falling back to default profiles");
 return Collections.singletonList(post);
 }
}

User’s fan data


@Service
public class FollowServiceImpl implements FollowService {
 @Override
 public List<User> getFollowers(Long userId) {
 try { Thread.sleep(1000L); } catch (InterruptedException e) {}
 int size = 10;
 List<User> users = new ArrayList<>(size);
 for(int i = 0 ; i < size; i++) {
  User user = new User();
  user.setUsername("name"+i);
  user.setEmail("email"+i+"@fox.com");
  user.setId((long) i);
  users.add(user);
 };
 return users;
 }
}

Note that each method sleeps for 1 s to simulate business time

We need to encapsulate another interface to assemble the data of the above three interfaces

PS: this kind of scenario is very common in work, and often the data we need to put together needs to be transferred to a third party through network request. In addition, some people may think, why not divide it into three requests? In fact, for the sake of the client’s network performance, we tend to transfer as much data as possible in one network request, provided that the data is not too large, otherwise it will be transmitted Time consuming will affect rendering. Many app home pages look complex, but there is only one interface in reality. All data is pulled down at one time, and client development is simple

Serial implementation

Writing a high performance interface is not only the technical pursuit of every back-end programmer, but also the basic demand of the business. In general, in order to ensure better performance, it is often necessary to write more complex code implementation

But everyone is lazy, so we often write serial call code like the following


@Component
public class UserQueryFacade {
 @Autowired
 private FollowService followService;
 @Autowired
 private PostService postService;
 @Autowired
 private UserService userService;
 
 public User getUserData(Long userId) {
  User user = userService.get(userId);
  user.setPosts(postService.getPosts(userId));
  user.setFollowers(followService.getFollowers(userId));
  return user;
 }
}

Obviously, the above code is inefficient. It takes at least 3S to get the result. Once the data of an interface is used, it needs to inject the corresponding service, which is troublesome to reuse

Parallel implementation

The aspiring programmer may immediately consider that there is no strong dependency between these data items, which can be obtained in parallel. It is implemented by asynchronous thread + countdownlatch + future, as shown below


@Component
public class UserQueryFacade {
 @Autowired
 private FollowService followService;
 @Autowired
 private PostService postService;
 @Autowired
 private UserService userService;
 
 public User getUserDataByParallel(Long userId) throws InterruptedException, ExecutionException {
  ExecutorService executorService = Executors.newFixedThreadPool(3);
  CountDownLatch countDownLatch = new CountDownLatch(3);
  Future<User> userFuture = executorService.submit(() -> {
   try{
    return userService.get(userId);
   }finally {
    countDownLatch.countDown();
   }
  });
  Future<List<Post>> postsFuture = executorService.submit(() -> {
   try{
    return postService.getPosts(userId);
   }finally {
    countDownLatch.countDown();
   }
  });
  Future<List<User>> followersFuture = executorService.submit(() -> {
   try{
    return followService.getFollowers(userId);
   }finally {
    countDownLatch.countDown();
   }
  });
  countDownLatch.await();
  User user = userFuture.get();
  user.setFollowers(followersFuture.get());
  user.setPosts(postsFuture.get());
  return user;
 }
}

The above code, changing serial call to parallel call, can greatly improve performance under limited concurrent level. But it is obviously too complex. If every interface writes such a code for parallel execution, it is a nightmare

Elegant annotation implementation

Familiar with Java, we all know that Java has a very convenient feature ~ ~ annotation. It’s just a black magic. You can realize very complex functions only by adding some annotations to classes or methods

With annotations and spring’s idea of automatic dependency injection, can we automatically inject dependencies and call interfaces in parallel through annotations? The answer is yes

First, we define an aggregation interface


@Component
public class UserAggregate {
 @DataProvider(id="userFullData")
 public User userFullData(@DataConsumer(id = "user") User user,
        @DataConsumer(id = "posts") List<Post> posts,
        @DataConsumer(id = "followers") List<User> followers) {
  user.setFollowers(followers);
  user.setPosts(posts);
  return user;
 }
}

among

  • @DataProvider indicates that this method is a data provider with data ID of userfulldata
  • @Dataconsumer represents the parameter of this method. It needs to consume data. The data ID is user, posts, and followers

Of course, the basic user information, user blog list and user fan data of the original three atomic services also need to be annotated


@Service
public class UserServiceImpl implements UserService {
 @DataProvider(id = "user")
 @Override
 public User get(@InvokeParameter("userId") Long id) {

@Service
public class PostServiceImpl implements PostService {
 @DataProvider(id = "posts")
 @Override
 public List<Post> getPosts(@InvokeParameter("userId") Long userId) {

@Service
public class FollowServiceImpl implements FollowService {
 @DataProvider(id = "followers")
 @Override
 public List<User> getFollowers(@InvokeParameter("userId") Long userId) {

among

  • @DataProvider has the same meaning as before, indicating that this method is a data provider
  • @Invokeparameter indicates the parameters passed in manually when the method is executed

Note the difference between @ invokeparameter and @ dataconsumer. The former requires the user to manually pass parameters when invoking at the top level; the latter is injected after the framework automatically analyzes the dependency and asynchronously invokes the result

Finally, we only need to call a uniform facade interface, pass data ID, invoke parameters, and return value type. The rest of parallel processing, dependency analysis and injection, are completely handled by the framework automatically


@Component
public class UserQueryFacade {
 @Autowired
 private DataBeanAggregateQueryFacade dataBeanAggregateQueryFacade;

 public User getUserFinal(Long userId) throws InterruptedException, 
    IllegalAccessException, InvocationTargetException {
  return dataBeanAggregateQueryFacade.get("userFullData",
    Collections.singletonMap("userId", userId), User.class);
 }
}

How to use it in your project

The above functions have been encapsulated as a spring boot starter and released to Maven central warehouse

Just introduce dependencies into your project


<dependency>
 <groupId>io.github.lvyahui8</groupId>
 <artifactId>spring-boot-data-aggregator-example</artifactId>
 <version>1.0.1</version>
</dependency>

And declare the scan path of the annotation in the application.properties file

#Replace with the package you need to scan for comments
io.github.lvyahui8.spring.base-packages=io.github.lvyahui8.spring.example

After that, you can use the following annotations and spring bean to implement aggregate queries

  • @DataProvider
  • @DataConsumer
  • @InvokeParameter
  • Spring Bean DataBeanAggregateQueryFacade

Note that @ dataconsumer and @ invokeparameter can be mixed and used on different parameters of the same method. All parameters of the method must have one of the annotations, and cannot have parameters without annotations

Project address and the above example code: https://github.com/lvyahui8/spring-boot-data-aggregator

Later stage plan

In the future, the author will continue to improve exception handling, timeout logic, solve naming conflicts, and further improve the plug-in’s ease of use, high availability and scalability

summary

The above is the whole content of this article. I hope that the content of this article has some reference learning value for your study or work. Thank you for your support for developepaer.