Using electron to build cross platform grabbing desktop program


Using electron to build cross platform grabbing desktop program

When talking about desktop application development technology, we will think of WinForm under. Net, JavaFX under Java and QT under Linux. These technologies are generally unfamiliar to web application programmers, because most web application programmers’ development skills are front-end JavaScript and back-end Java, PHP and other languages
What if web application programmers want to develop desktop applications? The learning curve of mainstream desktop application development technology is not low, and it is difficult to get started. The emergence of electron brings good news to web application programmers

Introduction to electron:

Electron is a cross platform desktop application development tool released by GitHub, which supports web technology to develop desktop application development. It is developed based on C + +, GUI core comes from chrome, and JavaScript engine uses V8

Simply put, electron platform uses JavaScript to connect UI and background logic. Background main process uses rich API of nodejs to complete complex and time-consuming logic, while UI process uses chrome to render HTML to complete interaction

I used springboot to develop a web application for grabbing mayor’s mailbox. Since there is no server to deploy, I now want to transplant the same functions to the desktop as a desktop application. For the development platform, I have the following requirements:

  1. Can use my existing technology stack: Web front-end JavaScript, server-side java or nodejs

  2. It can be compiled into DMG installation program under Mac and exe file under Windows platform

As a development platform, electron can meet my needs. After one day’s exploration, I completed this desktop application and finally packaged the DMG installation file on Mac platform

Using electron to build cross platform grabbing desktop program

Here’s how I used the electron platform to develop this desktop application

Review: mayor mailbox email capture web application

Before I start, I’ll first analyze the previous crawling web application. Its architecture is as follows:
Using electron to build cross platform grabbing desktop program
The application can be divided into four parts

  1. And get the content of the request to the Java HTTP class library

  2. Database: implemented by mysql, used to save the web page content after crawling, and provide search and query service

  3. Static interaction page: a simple HTML page, using jQuery to initiate Ajax and back-end interaction, and using handlebar as a presentation template

  4. Communication: use springboot to provide the necessary API (search service, full capture and update mail)

Design: build grab desktop application with electron

The desktop application to be implemented also needs to complete these four parts of work
Using electron to build cross platform grabbing desktop program
The main process of electron completes the functions of web page crawling, data storage and search by virtue of the rich ecosystem of nodejs, while the UI process completes the rendering of the page

  1. Grab program: use nodejs request, cherio, async to complete

  2. Database: using nedb storage under nodejs, as an application embedded database, can be easily integrated into desktop applications

  3. UI: Using HTML and front-end JavaScript class library to reuse static pages in previous web applications

  4. Communication: use IPC provided by electron to complete the communication between main process and UI process

Implementation: use electron to build grab desktop application


There are tens of thousands of mail messages in mayor’s mailbox. Due to the asynchronous feature of JavaScript, thousands of concurrent requests will be written by people carelessly. In a short period of time, a large number of attempts to establish a connection with the target server will be rejected by the server, which will cause the grabbing process to fail

  1. TCP connection multiplexing

  2. The concurrent frequency is controllable

I use the following three nodejs components:

  1. Request HTTP client uses the HTTP keepalive feature of the underlying nodejs to realize the reuse of TCP connections

  2. Async controls the concurrency of requests and the ordering of asynchronous programming

  3. Cherio HTML parser

code: crawlService.js

//Use request to get the content of the page
request('', (err, response, body) => {
    if (err) throw err;
    //Using cherio to parse HTML
    var $ = cheerio.load(body),
        totalSize = $('div.pages script').html().match(/iRecCount = \d+/g)[0].match(/\d+/g)[0];
    //Async is used to control the concurrency of requests and grab the contents of mail pages sequentially
    async.eachSeries(pagesCollection, function (page, crawlNextPage) {
        pageCrawl(page, totalPageSize, updater, crawlNextPage);

2. Realization of database

There are many choices for content storage after grabbing

  1. text file

  2. Search Engines

  3. database

Although the text file is simple to save, it is not conducive to query and search
Search engines generally need to be deployed independently, which is not conducive to the installation of desktop applications
Independently deployed databases have the same problems as search engines, so connecting to external MySQL is not used here

To sum up, I need an embedded database. Fortunately, nodejs has rich components. Nedb is a good solution. It can save data in memory and disk at the same time. It is a document embedded database, and uses mongodb syntax for data operation
code: dbService.js

//Establish database connection
const db = new Datastore({filename: getUserHome()+'/.electronapp/watcher/12345mails.db', autoload: true});
//Insert data using nedb
db.update({_id: mail._id}, mail, {upsert: true}, function (err, newDoc) {});
//Using nedb for mail query
Let match = {$regex: Eval ('/' + keyword + '/')}; // keyword matching
var query = keyword ? {$or: [{title: match}, {content: match}]} : {};
db.find(query).sort({publishDate: -1}).limit(100).exec(function (err, mails) {
    event.sender.send ('search reply ', {mails: mails}); // process the query results

3. Implementation of UI

The project directory of desktop application is as follows:
Using electron to build cross platform grabbing desktop program

I put the UI page in the static folder. The front-end UI development in electron is the same as the normal web development, because the UI process of electron is a chrome process. When electron is started, the main process will execute index.js Documents, index.js The application window is initialized, the size is set, and the UI entry page is loaded in the window index.html .
code: index.js

function createMainWindow() {
    const win = new electron.BrowserWindow({
        width: 1200,
        height: 800
    }); // initial application window size
    win.loadURL (`file://${__ dirname}/static/ index.html `); // load the page in the window
    win.openDevTools (); // open the devtools of chrome
    win.on('closed', onClosed);
    return win;

In the process of UI page development, it should be noted that jQuery, require and other components will fail to load by default. This is because the browser window loads some methods of nodejs, which conflicts with the methods of jQuery class library. Therefore, we need to do some special processing to delete these nodejs methods in browser window
code: preload.js

//Solve the problem that jQuery and other components are not available due to require conflict
window.nodeRequire = require;
delete window.require;
delete window.exports;
delete window.module;
//Solve the problem that the chrome debugging tool dextron is not available
window.__devtron = {require: nodeRequire, process: process}

4. Realization of communication

In web applications, the communication between pages and services is carried out through Ajax, so our desktop applications can’t also communicate in the way of Ajax? Although this theory is feasible, there is a big drawback: our application needs to open an HTTP listening port. Generally, personal operating systems prohibit software from opening http80 port, Opening other ports can easily cause port conflicts with other programs, so we need a more elegant way to communicate
Electron provides the interface between UI process and main processIPCAPI by usingIPCThrough communication, we can realize that UI page can send query and grab request to nodejs service logic, and nodejs service can actively inform UI page of update of grab progress
Using electron’sIPCIt’s simple.
First, we need to use ipcrenderer in UI to send messages to the custom channel
code: app.js

const ipcRenderer = nodeRequire('electron').ipcRenderer;

//Submit query form
$('form.searchForm').submit(function (event) {
    ipcRenderer.send ('search-keyword', $(' input.keyword '. Val()); // initiate a query request
ipcRenderer.on ('search reply ', function (event, data) {// listen to the query results
    if (data.mails) {
        var template = Handlebars.compile($('#template').html());

Then, we need to use ipcmain in the nodejs code executed by the main process, listen to the channel defined before, and then we can accept the request from the UI
code: crawlService.js

const ipcMain = require('electron').ipcMain;

ipcMain.on('search-keyword', (event, arg) => {
  ... // process query logic

ipcMain.on('start-crawl', (event, arg) => {
  ... // process grab logic

Desktop application packaging

After solving the above four problems, the rest of the program is easy to write. After debugging the program, using electronic builder, you can compile and package the executable files for different platforms